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ABSTRACT . , • 

The -Program Standards Evaluation System was 
in response 'to evaluation requiremeats in the 1973 Retlabilitation 
Act. The system includes procedures for using standards data to 
monitor and evaluate vocational rehabilitation (VR> service outcomes t 
and outputs as well as standards on /^y procedural issues. This 
trainee handbook has been prepared |or persons receiving training in 
the use of the Rehabilitation Services Administration-* s Vocational 
Rehabilitation Program Standards Evaluation System and in collection' 
of ^information for performance and procedural standards . The 
materials have been designed to b? used in conjunction with verbal 
and visual presentation materials. This hand*book contains ati 
introduction to the Program Standards Evaluation System and its uses^r 
Organized into five chapters, it is intended to provide the reader 
with an overview of the system's individual components as well as the 
system's overall logic. Individual chapters provide fuller detail on 
the. conceptualization and implementation of specific aspects of the 
system. The five chapters cover the following topics: (1) an overview 
of the Program Standards System;' (2) computing and presenting the 
eight performance standards; (3) administering the closure and 
followup sui/veys; (4) the fi^ve procedural standards; and (6) an 
introduction to the modified case review schedule. (KC) 
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PREFACE 

These materials have been prepared for ^individuals receivinj^ 
training in the use of the RSA VocationaT Rehabilitation Program Stan- 
dards Evaluation System and in coMection of information for performance 
and procedural standards. The materials have been designed to be used 
in conjunction with verbal and visual presentation materials. More 
detailed informat^ion on the standards system can be found in two earlier 
Berkeley Planning Associates' reports, the Program' Standalrds Analytic 
Paradigm (Berkeley, California: June 11, 1982) -and the Program Standards 
Guidance Materials (Berkeley, California: November 17, 1981). 
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INTRODUCTION a 

The 1973 Rehabilif ation Act Contained, among its many other provisions, 
a requirement that evaluation standards be devised and implemented to 
measure the performance of the VR^^rrogram in achieving, its mandate. Over 
the last four years, Berkeley Planning'Associates, under contract to tKe 
Rehabilitation Services Administration, has developed a revised system of 
evaluation standards. Two distinct sub-systems of performance measures 
were developed. One, the proposed Program Evaluation Standards, evaluates 
the federal-state VR programs. The other, the proposed ^rc3j feet Evaluation jf 
Standards, measures the effectiveness of individual projects, as well as 
aggregated program authorities funded by RSA discretionary funds. During 
the la3t three years, these systems were pretested in six model state evalu- 
ation, units. In .this pretest, BPA assumed the responsibilities which will 
ultimately he under the authority, and perhaps ^ctual^ execution, of RSA, )fy 
providing ^training in the instruments and procedures for their administra- 
tion, providing technical assistance to the states in conducting the pretest 
and analyzing their data, and providing the basic reporting of the states' 
performance. 

The final recommended Program Standards consist Of eight ^Performance 
.standards and associated d^ta elements; and £ive Procedural Standards and 
associated data elements (see Table 1). The Perforinance Standards pertain 
to service outputs ajid outcomes (e.g^. , coverage, effectiveness, ilnpact) > 
while the Procedural Standards pertain to service method and process (e.g., 
case handling). -If implemented, the Program Standards would require a 
revisjed reporting, s^st^m for all state agencies. The^ federal administration 
would, in turn, ^generate information for measuring, the achievement of overall 
program goals and' for monitoring key processes which protect client interests 
In addition, the pystem design ioicludes a mechanism for analyzing and under- 
staTWing the fac^J^rs contributing to *goai^chiev€fnent, and for applying that 
understanding in feupport of federal and st^te program managers and policy 
makers in de^isioh^ regarding changes in program procedures and policy/ state 
needs for technical assistance, and program needs for further investigation. 
' \ j • 

i' * ♦ 
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, , . • Table I 

VR Program Standards and Data Elements: Final Recommendntions . 198 1' 



PERFOR.M.A-NCE ST^S'PARDS \SD DATA ELEMEJfFS 

— 

1 . Coverage * 
VR shall ser\e the maximum proportion of the potentially eligible target population, subject 
to the level of federal program funding and priorities among clients. 

(i) Clients served -per 100,000 popul^ition 

(ii) Percent severely disable4 served 

2. Cost-iigfect;veness and Benefit-Cost Return 

The VR program s^all use resources in a cost-effective, manner and show a positive . return to 
society of investment in* vocational rehabi litat*ion of disabled clients. 

' ' S • 

(^1) Expenditures per competitively employed closure 

(ii) Expenditure per 26 closure * 

(iii) Ratio of total VR benefits to total VR costs (Benefit-cost ratioV^ 

(iv) Total net benefit from VR services (Discounted net present value) 

3. Rehabilitation Rate # / 

VR shall maximi:e the number and proportion of clients accepted for services who are 
successfully rehabilitated, subject to the meeting of other standards. 

(i) Percent 26 closures 

(ii) Annual change in number of 26 closures « 

4. Economic Independence 

Rehabilitated clients shall evidence economic independence. 

' (i) 'percent 26 closures^ with weekly ea'mings at/abpve federal minimum wage 
. (ii) Comparison of eanyngs of competitively employed 26 closures to earnings of employees 
in state ' ^ 

5. Gainful Activitv 
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There shall be maximum placement of rehabilitated clients into competitive emplo>ment. 
Noncompetitive closures shall represent an improvement in gainful Activity for the client. 

(i) Percent' 26 closures competitively employed ' ' • ^ 

(ii) Percent competitively employed 26 closures with hourly earnings at/above federal 
minimum ^<jigs 

(ill) Percent noncpmpetitively employed 26 closures shouing improvement in function and 
life status (implement after FAI/LSI pretest) 

« 

Client 'Change 

Rehabilitated clients shall evi4ence vocational gams. 

(i) Companion of earnings before and after VR services 

(ii) (In addition,, changes in other' statuses , and functioning ability, when such measures 
become available) ^ 

Retention^ 



Rehabilitated clients shall retain the benefits of VR services. ^ - 

(i) 'Percent 26 closures retaining earnings at foUou-up 

(ii) Comparison of 26 closures with public assistance as primary soiifce of support at 
closure and at follow-up , 

(III) Percent noncompetiti vely employed 26 closures retaining closure skills at t'olh>u-up 
(implement after FAI/LSI pretest) 

S, Sat IS fact ion ^ 

Clients shall be satisfied uith the VR program, and rehabiUtated clients shall appraise 
VR servicos as useful in achieving and maintaining ^their vocational objectives. 

(i) Percent closed clients satisfied with o^'crall VR experience 

(ii) ^Percent closed clients satisfied uith: counselor, physical restoration, 

)ob training services, pl&cement services 

(iii) Percent 2b closures Judging services received as useful in obtaining their .lob/ 
homemaker situation or in current performance ^ 
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Table 1 (cont.) 



PROCEDUR.^L STANDARDS , 

9. • R'500 Validity 

Information collected on clients by the R-300 and .all data reporting systems used by RSA' 

shall be valid, reliable, accurate, and complete.^ 

10. Eligibility \ . ' 

Eligibility decisions shall be based on accurate and sufficient diagnostic information, 
and VR shall continually review and evaluate eligibility decisions to ensure that 
decisions are being made in accordance with laws and .regulations . 

11. Timeliness 

VR shall ensure that eligibility decisions and client iroveraent through the VR process 
occur in a timely manner appropriate to the needs and capabilities of the cliei^ts. 

12. IWRP 

VR shall provide an Individualized Wri tten Rehabilitation Program for each applicable 
client and VR and the client shall be accountable to each other for complying uith this 
j agreement. 

I 15. Goal Planning ' ' 

Counselors shall make an effort to set realistic goals for clients. Comprehensive con- 
sideration must be given to all factors in developing appropriate vocational goals such - 
that there is a maximum of correspondence between goals and outcomes: competitive goals 
should have competitive (jutcomes and noncompetitive goals should have noncompetitive 
outcomes. 



This Trainee Handbook is designed as an introduction to- the Program 
Standards Evaluation System and its uses. It is intended to provide the 
reader with an overview of the system's individual components ^as well as 
the system's overall logic. Individual chapters have been developed to 
•provide fuller- detail on the conceptualization and implementation of spec 
ific aspects of this system. 
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' . _ AN OVERVIEW OF THE 

- PERFORMANCE STANDARDS SYSTEM ' . 

DESCRIPTION OF THE VR STANDARDS: HISTORICAL BACKGROUND 

The Federal -State Vocational Rehabilitation (VR) Program provides 
resources to disabled persojis who confront handicaps, and have vocational 
potential. Th^ institutionalization of the VR-*^rogram,, which became law 
in 1920, was assisted by the compelling economic argument that a self- 
supporting citizen was pr^fera|ble, in terms of the national welfare, to a 
disabled persoh who was dependent" upon public support. ' Initially, the , 
legisTation was concerned with providing the physically disabled the medical 
services neces-sary for them to find jobs. In subsequent amendijients to the 
legislation, the scope of eligibility and services was expanded to include 
services to the* family o£ the handicapped and to cases' of psychological . 
disorder, alcoholism and drug abuse. In addition, the Rehabilitation Act 
of 1973 inicluded a mandate to serve the severely disabled, those with the 
most handicapping conditions* and in need of more intensive services. 

Consistent with' the historical emphasis on employment, the success of 

seVvice to a client has been 'measured by Whether or not the ^client is ''closed 

rehabilitated'i. or placed in a work^ Situation for at least, 60 days after 

closure. Competitive employment has been the favored placement,, but success 

may also be claimed for placement in * sheltered employment or "in a homemaker 

or unpaid family worker situation. These last two options are considered • 

^successes be.cause performance of these roles may free other family members to 

enter the worjc force. 

•The 1973 Act also .contained a provision calling for the- development and 

use'of program performance standards. Specifically, the A^ "provided' that : 

**The Secretary shall develop and publish general standards 
fcxr evaluation of the programs and project -effectiveness 
^ in achieving the objectives of this Act.*/' [P.L. 92-112, 

Section 401 (3) (6)] * . • . 



.'^ The first standards, pub lishe4 in 1974, were .prepared by RSA and 
reviewed by members of the Council of State Vocational Rehabilitation , ' 
Administrators (CSAVR). fhes^ standards ' identified a number of features , 
in the rehabilitation process* The perforiflance of each state agency " 
was to be compared against other agencies ♦ States would learn about 
their comparative perfoinnance after each state result was included in , 
the st;andards analysis. 

For standards which focused heavily on compliance with the spirit 
and management of the rehabilitation process, dat;a elements or Statistical 
measures which drew upon regularly reported client and program data were 
identified. The norm for performance on most elements wajb^et as plus or 
minus* one s;J:andard deviation from the mean performance of all state VR 
agencies. 

\Vhile the- original set of standarfls met the requirements for reporting 
set forth in the Xct, RSA sought further refinement in the system. A' second 
developmental activity was supported to build upon the state's experiences 
with the first standards in order to develop a better system. In 1975, 
RSA contracted with the Urban Institute tg us,fe a much more analytical 
approach to ref inking th^ standards.. The Institute had proposed the develop- 
ment of a simulation model of the rehabilitation system, and the ultimate 
setting of standards performance levels based upon an analysis using the 
model. 

In its final reports; the Institute criticized the existing standards 
system and recommended". development of sophisticated statistical techniques 
needed, for comparison of state programs leading to a comprehensive micro- 
simulation or "overall evaluation framework." The Institute effort resulted 
in focusing RSA and state attention on a number of conceptual and analytical 
problems and issues inherent in the. clevelopment of standards. -JEt did not,' 
however, result in changes in the/ex4.sting standards. 

In the* Fall of 1976, RSA a:gain calle^^for further development and' . . 
refinement of the standar4s through a contract with Berkeley Planning 
Associates' (BPA). This work called for a, new cojjceptual approach to the 
development and refinement of the standards. 

The states' reaction to the 1974/75 standards pointed to the need 
for reassessing the content and purpose of a performance standards system 
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for rehabilitation-. BPA's new design effort b^gan with an ejcamination of 
alternative conceptual approaghes to ths development Df standards* A review 
6f standards -setting in 'other 'social service fields'^ showed a variety of 
approa'ches^ from; -a focus cyf inputs ' (either as structural or "gate-Keeping" 

. eligibility standards)»vto proc'e;S^es .' (measures of t'best practice") to 6utcomes ' 

' (or program impatts). Af ter'artalysis of the strengths' aiid ^eakn^sses of 

^ these al^ernativ^^ approaches,^ an appr.oach, which •emphasized program outcome 
was recommended. - In 1978, BPA developed- a. set of revised pferformance 

^tandards^ focusing on several measures- of rehabilitation success. In addi- 
tion, several st'andards '"for procedure were defined. 

Following the development of the. revised standards, BPA conducted ^n 
extensive pre-telt of the systell^^4n six sample states* This experienoe, , 
as well as a cajre^l review ofthe systen) by nuiiterous expejts in the' field, 
reSttlted in another wave of ?'rpvisions in 1981. . - 

IkV BPA's final, recommended standards and dat^ elements ^^for^measuring 
and monitoring theiy acMevemerit are shown in Tabie 1 . This table com- ► 

, pares these standards and, data elements to the BPA "proposed standards of 
1978 and to the existing standards promulgated and in use since the mid-1970s 

.A close examination of the table will reveal that the changes between , the 
two BPA recommended sets of standarrds are relatively small and technical, 
as compared to, the differences -between the BPA standards and the existing ' 
standard?. Reviewed individtially , the standards are as follows; . 

# The first standard addresses^ coverage, or the extent to which the 
vocational rehabilitation program' is serving the eligible ^target population. 
The need to ensure accessibility of services to all the eligible disabled 
is of paramount importance to RSA and the states. The first data element , 
-- clients served per 100,000 population provides a proxy measiire of 
coverage of eligible population. The second measure — percent of clients 
served who are severely disabled |)fteasures achievement of the priority, 
legislated for the seyerely disabled by Congress.* 

V ' . ' : ^ 

' ' . * . ' ' 

^VR' Pr ogram Evaluation Standards: >A Critique of the State of the Art. 
Berkeley Planning Associates, Berkeley, California, January 7, 1977, pp. 
42-48. * ^ ' • . ' 

/ ^ ^ Alternative Conceptual Approaches to- Standard s (Working Rape*- #2) , 
Berkeley Planning Associates, Berkeley, California, Mai|ch 25, 197^.-- 
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Table 1 ; 

Evolution of VR Program Standards 



r>i>l Kc\ iSvd bPA ^t.tnJ.irds 



1. VR sliall serve tlie n*.ivjnfiiin pi^opor- 
tion of the putciit iall)'^liyil)le 

'^^largct population, siil>jt:ci to the 

level of fodeml program funding 

and priOMties arr»oii^ clients • 
/» ' . 

(i) Clients 'served per 100,000 
p5pu>ation • 

(ii) Percent severeX^ disabled 

( 4 . 



Tlie \'\{ prograjn shall use resources 
in a ^cost-effective marque r and show 

,a positive retujn to society of 
iiivcstnicnt in vocatifnial rehab^li- 

Ota t ion of dis,tbled cJie;its * 

(i) ^ Expenditures per competi- 

tive ly 'eir.ployed -closure 

(ii) ^ TxpeiKh ture ^er 26 closure 
(lii) Benefit -cost ratio 

liv) Discounted not present ^ 
.value ■ t 
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VR shall mciAW J 2c tl\e number and 
proportion of clients accejited for 
services' who are successfully re- ^ 
habilitated, subject to the 
meeting of other standards 

Percent closures 
Armual change in niuriber of 
26 closures ' 



(lor 



r 



1976-1978 m\ Standards 



VR shall serve the maximum proj^r- 
tion of the poteiiyially eligib^,c 
target population, subject to the 
level of federal program funding 
and priorities arnong clients 

(i) Corqiarison of caseload served 
' to expenditures » 

(ii) Clients servejl per 100,000 
» * population 



The VR iTjl)gram shall, use resfourceS 
in a cost-effective manner and show 
a positive return to society of 
investment in vocational rehabili- 
tation of disabled clients 



(i) 

(ii) 

(iii) 

(iv) 



Expenditure per* competi- 
tively employed closure 
Expenditure per 26 closure 
^Benefit-cost ratio 
Discounted net present 
value 



1975 General Standards for evaluation 



tt^.?haU maximize the number and 
proportion of clients accepted for 
services who are successfully re- 
habilitated, subject to' the ' 
meeting of other standards 

(i\ ' , Percent 26 closures 



To ifisure tiiat the rehabilitation pro^jram is serving the 
eli^N^le disabled population 5nd that these services are 
prov^pd in an:' equitable manner 

(i) Estimate of the total population eligible for VR services 

(ii) Number of accepted cases served (statuses 10-30) for 
the year * ^ 

(iii) -Percent of annual increase or decrease in number of 

accepted cases served (statuses '10-30) * 

(iv) Number of cases closed rehabilitated during the year 
(status 26) 

(v) Accepted cases (statuses 10-2^) as a percentage of the 
*' total of cases closed not -acceptedr (statlis £)8) plus 

those cases^ accepted (.Statuses 10-24) 

' * * 
To insure that ^available resources are utilized to achieve 
maximum- operational efficiency 

(i) • Average/'case service cost per aoi^cpted case closure 

(statuse^ 26, 28 and 30) where case service cost W 
the statue vocational rehabilitation agency was 
involved . * e 

(ii) Percent of clients, receiving rehabi li tat ion ''services 
at no cost to the vocational rehabilitation agency 

(iii) Percent distribution of. total vocational rehabili- 
tation dollars spent^for agency operations each 
fiscal^year as reported on the RSA-2 expenditui;^ 
report * 

(iv) Average case service cost per type of vocational 
rehabilitation case service involving cost to the 
state agency, and percentage of individuals re- 
ceiving specif ic 'vocational rehabilitation service j 

To insure that rehabilitated clients are placed in gainful 
employment suitable to their capabilities* 

(i) Percent of those placed in competitive qmployment 
(wage and salary earners and self-employment) 

(ii) Percent of those places in noncompetitive employ- 
ment (sheltered workshop and others) 

(iii) I'ercent of those placed as )\pniomakei's 

(iv) Percent of those places a$ unpaid family worker 

(v) P,ercent of those placed in business enterprise program 



> 
I 



Tliis data element does not appear in th6 1979 analysis. 
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Table 1 (contihued) 



Performance ^tanJurJs (cantinujed) 



4. Rdubil it.iteJ clients shall evi- 
dence eoonoraic inJcpc-ndence 

(i) Percent 26 closures with 
weekly earnihgs at /above 
federal miniirjuin wage 

(ii) Conpari^on of earnings of 
competitively eir.piqyed 26 
closures to eanungs of 
employees' in state 



1976-1978 BPA Standards 



4. Rehabilitated clients shall evi- 
dence increased economic indepen- 
dence 



M " 
There shall be maximum placcincnt 
of rehabi li t.ite^J clients into 
coinpctitive cwployment Noncom- , 
petitive closures shall represent 
.an impiovec-ent in gainful activity 
for the client 

(i) Percent 26 closures com- 
petitively employed 

(ii) Percent compet i t i ve ly em- 
ployed 26 closures with 
hourly earnings .it/above' 
fxrdejral mininiun wage 

(l i'i) Percent noncoff.pet i t i vely 
• employed 2o closures 

showing improv'crtcnt in * ' 
Function and life status 



Rehabilitated clients shall evi- 
dence \ocatiohal .gdins (client 
change) 

(i) Comparison of earnings 
before and .ifter VR services 

(ii) (In nddition, chaiigos in 
other statiise^i, and func- 
tional ability, when such 

measures become available) 



(i) 
(ii) 

(iii) 
(iv) 



Percent 26 closures with 
weekly earnings at/above 
federal minimum ' • 

Comparison of earnings of 
competitively employed 26 
closures to earnings of 
employees in state 
Comparison. of earniflgs be- 
fore and after VR services^ 
Comparison of 26 closu-res 
with public assistance as 
primary source of support - 
before and after VR services 



Tliere shall be maximum ]>lacement 
of rehabilitated clients into 
competitive employment. Nonoom- 
.petitiVe closures shall be in 
' acc*ordance with the IWRP goal and 
shal hrepresent an improvement in 
. gainfin^ctivity for tlie cfient, 

(i) Percent- 26 closures com- 
petitively employed 

(ii) ' Percent 26 closures with 

hourly earnings at/above 
federal minimum wage 

(iii) Percent noncoinpetitively 
employed 26 closures showing 
improvement in gainful 
activity 

(iv) Percent 26 closifres with com- 
petitive outcome's or with 
noncompetitive Outcome and . 
noncompetitive goal 



1975 General Standards for Evaluation* 



2. To insure that rehabilitated clierrts are placed m gainful 
employment suitable t|||pthcir capacities 

(vii) Average weekly earnings in the week before referral 
of all rehabilitated clients, including clients with 
zero^^iomings 

(viii) Avera^ weekly earnings at closure of all rehabilitated 
clients, includirtg clients with zero earnings 



To insure that rehabilitated clients are placed in gainful 
employment suitable, to their capabilities ^ 



(i) ^ Percent of those placed. in competitive eraplo/ment 
(wage and salary earners and self-employment)* 



0 
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Table 1 (continued) 



t^erformance 5^tandards (continued) 

^ 

^ -113 
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Rehabilitated clients shall retain 
the benefits of VR services 

(i) Percent 26 closures retain- 
ing enrnings at follow-up 

(ii) Comparibon of 26 closures 
with public assistance as ^ . 
prinjry source of support 

ul closure nnd at follow- 
up / 

(ill) Percent noncompetitive ly 
ejf.ployed 20 ensures re- 
taining c-losurc skills at 
fol low-up 

Clients shall be satisfied Vith 
the \R program, and rehabilitated 
clients shall ai)praise VR services 
as useful in achieving and main- 
taining their vocational objec- 
tives • » 

(i) Percent closed clients 
satisfied witii overall VR 

^ experience » 

(ii) Percent closed tlieitts satis 
ficd with: information pro- 
vided, counselor promptness, 
pliysical re^vtoriitlon, job 
training services, place- 
ment services \ 

(iii) Percent 26 clo';ures judging 
services received as useful 

, . in obtaining their job/Iiome- 
maker situation or in cur- 
. ' rent perforaiance 



1976-1978 BPA Standards 



6. 



7. 



Vocational 'gains shall be attri- J 
butable to VR services (causality) 

(i) Comparison of earn'ings 
change from referral to 
closure of 26 closures - 
'to e'arnings change of a 
• control group 



Rehabilitated clients shall retain 
the benefits of VR services 



7^ 



(ii) 



(iii) 



\ 



Percent 26 closures retain- 
ing earnings at follow-up 
Comparison of 26 -closures^ 
with public assistance as 
primary source of support 
at closure and at follow- 
up 

Percent jioncompctiti vely , 
employed 26 closures re- ' 
taining closure skills at 
follow-up 



8. Clients, shall be satisfied with " 
the VR program, and rehabilitated 
clients shall appraise VR services 
as useful in athieving and main- 
taining their vocational ^jec- 



(i) 



(ii) 



(iii) 



(iv) 



Percent closed clients 
satisfied with ovprall VR 
experience 

Percent closed clients 
satisifcd with specific 
aspects of VU 

Percent 26 closures judging 
services received to have 
been usc^l in obtaroing 
tlieir job/homemaker situa- 
tion 

Percent 26 closures juUging 
services to be useful /In 
current |)crform,'i;ice ou the 
job/homemaker sitiia|p 
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To irisure that client^ closed rehabilitated retain the benefits 
obtained from the rehabilitation procxTss 



(i) 



(ii) 



(iii) 
^iv) 



Percent of rehabilitated events still employed at time 
o/ follow-up, specifyingyOne year,' two years, or three 
years after closure ^ 

Percent* with eqmings at follow-up, mean earnings at 
fol low-up 

Percent increase or decrease of earnings at follow-up 
Percent of rehabilitated clients (status 26) unemployed 
at follow-up for: less than one month, one to throi < 
months, four to six months, seven to 1*2 months, m^c 



than 12 months 



9. To insure that the clientjis satisifed with the vocational 
rehabilitation servi^^gs^s developed with the counselor 

(i) Percent. of clients rehabilitated throughout the fiscal 
year (status 26) and not rehabilitated (statuses 28 plus 
30) throughout the fiscal year who express satisfaction 
with the following, sppcifying one year, two years, or 
tliree years : • / ' " • 

Ca) counselor's willingness to listen to client^s ideas 

and suggestions in developing the IWRP 
.0>)^ adequacy' of information .l)rovided by counselor to 

clients foif understanding their disability 
(c) promptness in the delivery of -services . 
-(d) kind o*f training received ' 
^(e) benefits of training reoci'Ved 

(f) assistance in seeking j<Jb and final employment 
'(g) results of physical restoration services ^ 

(iv) Percentage of clients contacted during the foUow-up 
period who stated they would rccoit«jiljnd vocational 
* reljabllitation to a disabled fricn/ 



Table 1 (continued) 



Procedural Standards* 



10 SI l^cviscd H^A StnndanS^s 



1976-1978 BPA Standards 



197S General Standards for [-valuation 
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9. 



10. 



11. 



Informati9n collected on clients 
by the R-500 and nil datu re- 
porting sy$te:ns used by shall 
be valid, reliable, accrui^ate, and 
complete 

Eligibility decisions shall be 
based oK accurate and sufficient 
diagnostic information, an4 VR 
shall continually review and 
evaluate eligibility decisibns to 
insure that decisions are being 
made in accordance with laws' 
and regulations 

VR shal 1 insure that eligibility 
decisions and client movement 
through the VR process occur 
in a timely manner appropriate 
to the needs and capabilities 
of tlie clients 



12. 



VR shall provide rrn individ- 
uali 2edMVri tten RoUabi 1 1 tat ion 
hrogram for each applicable client 
*and VR and the client shafl be 
accountable to each other for com- 
plying with this agreement 



9. 



10. 



11. 



Information collected on clients 
by the R-300 and all data re- 
'porting systems used by PSA shall 
be valid, reliable, accuragc-, and 
complete ( 

Eligibility decisions shall be 
based on accurate and sufficient 
diagnostic information, and VR 
shall continually review and 
evaluate e];,igibility decisions to 
insure that decisions are being 
made in accordance »with laws 
and regulations • 

VR, shall insuud^jhat eligibility 
decisions and client movement 
through the VR process occur 
in a .timely manner appropriate 
to the needs and capabilities 
of the clients 



12. 



VR shijjl provide an individ- 
ualized Written Rehabilitation 
Program for each' applicable client 
and VR and the client shall be 
accountable to each other for com- 
plying with this agreement 



To insure that undue delays are avoided in providing clients 
with VR services ' 



(ii) 
(iii) 

Civ) 

0 

(V) 

(vi) 
(vii) 



Average time 'from combined referral-applicant statuses 
(statuses .00-020 to closed not accepted (status 08) 
Average time 'in extended evaluation (status 06) for 
c6ses closed not accepted (status 08) 
Average time from combined referral-applicant statuses 
(statuses 00-02) to accepted statuses for cases closed 
rehabilitated (status 26) and closed not rehabilitated 
(statuses 28-30) during the fiscal., year * 
Average time in extended evaluation (status Obj ^or 
dases closed rehabilitated (status 26) and closed not 
rehabilitated (statuses 28 and ZOj <Juring the fiscal 
year * > 

Average time from accepted ctise statuses (statuses 10-^ 
24) to clos*ed rehabilitated (status 26) 
Average time in accepte<< case statuses XstatuseS 16- 
24) to closed not reljabili'tated after Rehabilitation 
program was initiated (stjtJus 28) ^ 
Average tinfe in accepted case -Statuses (statuses 10- 
24J to closed not ithabilitated before the rjchabili- 
tation iprograjn was initiated (status 30) 



•IroLcdnr.tl St.indtifdi art nut niLa.su'red with data elements, but through case review and use of designed instruments. For more information, please 
•See BPA's Report on the (*retest of the Revised Vocational Rehabilitation Program Standards , Volume 2, Draft 16 July 1981. 
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P)SI lU'vi -..'tl I I'A Sr iiuJ.irtls 



t iul.irtis j ' 



1976-1978 BPA St^iiuLirds 



1975 General Standards for f.valiKition 



5, T9 i ns\ire 'that manag^ble-sizcd caseloads arc j||.iintainud 



(I) 

(ii) 

(iv) 
Cv) 



Ntimber of caseload carrying x;oijn.selor iruin years 
Number of authorized and funded fuU-time, caseload 
carrying counselor positions > $ 
Number and percent of rehabilitation counselor turnover, 
i.e., hiring rate and separation rate 
Average size of caseloads as of ScptenA;er 30 per number 
of authorized and funded caseload carrying cOuJiselor 
positions ii ' 

Describe the. process, if any, ployed by the state 
for each of the following functions: caseload manage- 
ment; caseload monitoring; caseload review 



7, To insure that the need for post-employment services is satis- 
fied 

(i) Percent of rehabilitated clients in the pravious fiscal 
year (status* 26) receiving post-employment (post- 
closure services during the 12 months following closure 

(ii) Percent receiving the following types of 'post- employment 
services of the total receiving post-employment services 

(a) diagnostic and evaluation 

(b) restoration (physical and mental) 

(c) , 'training ^ 

(d) guidance and counseling only 

(e) inaintenan\:e 

Xf) transportation ^ 
(g) other ^ 

8. To insure that agencies are consistenly identifying rettsoos wlity 
clients are not successfully rehabilitated 

(i) Percent of status 08,28, and 30 closures -by .the f 
following reasons: 

(a) tunable to locate or unable to contact or moved 

(b) handicap too sevjere .or unfavorable medical 
prognosis 

(c) refused services or* further services 

(d) death . , 

(e) client institutionalized 

(f) ' transferred to another agency 

(g) failure to cooperate 

(h) no disabling condition .(08 closure only) 

(i) no vocational h^andicap (08' closure only) 
(j) other 

(ii) Cases closed not .rehabilitated (statuses' 28 pfus 30) 
as a percentage of the total accepted cases closed 
(statuses 26 plus 28 plus 30) 



> 
i 

00 
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19vSl 'Revised BPA Standards 


1976-197« BPA Standards 


■ 4 ! 

197S General Standards for Evaluation 


13. Coim^Llors shall z\aVq an effort' 

to .set I ca fistic goalb for clients. 
ComprchenbxU'O consideration must 
^bc given to all faciois in ^cvel- 
oj)ing apj)ropriaie vocational goals 
siich that there is a maxinium of 
. . correspondence beiueen goals and , 
outcomei>; competitive goals should 
have competitive outcomes and non- 
coRjpecitive go.ilb sliould have non- 
competitive outConv^s.' 




2, To insure that rehabilitatc^d clients are placed in gainful ^ 
employment suitable to theii'r capabilities 

(vi) Those who received training related to the job family 
in which they were placed (as identified by the first 
digit of the Dictionary of Occupational Titles code) 
as a percentage of the total number who received 
training 
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• The second standard addresses directly the cost-effectiveness of 
the state program's overall use of resources , and the benefit-cost returns 
from investment in vocational rehabilitation services . The first two data 
elements measure the coSt* of achieving desirable .outcomes first, expend- 
itures per competitively employed closures, and second, the expenditure per 
26 closure. The focus on- competitively emplpyed closbres recognizes the 
policy decision in RSA that such a closure is^ the p^T>gram^s hrf^est priority. 
The second two data fiements focus on the two accepted measures of Benefit- 
cost returns the benefit-cost ratio and discounted present value, the 
benefit-cost model developed at Berkeley knd used over the years by RSA 

and many state agencies for' reports to Congress and state legislatures, 
and which was favorably reviewed by many independent specialists, is the 
model used to generate 'these summary data elements. The model is 'to be 
expanded by incorporation of subsystems being developed by the Texas Institute 
for Rehabilitation Research (TIRR) for taking the less monetary benefits of 
increased functional capacity and other aspects of independent living into' 
account . 

• The .third standard monitors the quality of service outcomes being 
achieved by the program and uses the traditional data elements of percent of 
closures which are successful [the. 26 closure) and annual change in the 
number of 26 closures. These data elements have a long history of use and 
acceptance in state programs as measures of how many clien^^^s VR is success- 
fully serving. . ' - . ' 

• The fourth standard focuses on whether rehabilitated client,s evi> 
dence increased economic independence , recognizing that VR's'most basic 
purposje is to assist disabled persons in finding gainful employment that 
will permit. their economic -self-sufficiency . Two data elements compare the 
wages achieved by rehabilitants to national standards (the minimum wage) 
and to 3ta^te jTOrms (earnings of employees in the state). These again are ' 
measures of the quality of service outcomes. • ^ ^ ' y 

• The fifth standard focuses on competitive and non-competitive 
employment outcomes in order to assess the quality of closures obtained by 
VR agencies. The first two data elements measure the percent of 26 closures 
who achieve competitive employment, and among these the percent employed at 
or above the national standards of the minimum wage.' The last data element 
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recognizes that competitive employment may not be ttie appropriate placement 
for all clients, but that it still is important that rehabilitation services v 
achieve improvements in gai^nful a|||ivity for those clients for whom emploj^- 
merit is not the goaT, For non-competitive* clq^ures, then, a data element 
measures the percent showing improvements in function and life status> The^ 
instrumentation for determining such improvements is b^ing developed 6y * ^ 
others for inclusion in the vMIS, and- will be pretested in subsequent years 
by RSA; ^ * ' 

• The sixth standard is directed at measuring client change before and 

after service* The 1978 standards recommended probing causal relatipnships 

between- services and out^romes and ^dging how much of the gain exhibited by 

clients is really attributable to the services they receive. This was because 

> 

both clients of VR and non-clients may show, over the same time g^^riod, 

^ • ' . A 

increased eamings, increased levels of skills, and other vocational gains. 

After pretesting a range of measures, including the, use of comparison groups 
of unsuccessful closures, data elements are recommended whiclv simply measure • 
**before-after** changes in eamings^nd^Cwhen MIS data becomes^ available) in 
functional capability. These measures are highly limited for" input ing caus- ,j 
alitv to VR service impact but they i^rovide some control for the client's 
capability prior to services. BPA has recommended that the mandat-e for , 
establishing causality be fulfilled through ^riodic controlled research' 
studies of clients on a national level as part qf supportive evaluation under 
taken by RSA. The complexity of such*research makes it infeasibjLe for ^ 
completion by state programs as part of their routine, ongoing evaluation 
activity. "Thus the focus' of the standard .becomes' one of measuring client 
change, r^er than establishing that the clie;nts' change .is due toJ^R 
^ services. ^ 4 % 

• ' The seventh. St anijard again monitors quality- of service outcome and 
overall program effectiveness, afid focuses on the retention of client* benefits 
from VR services over' time . The dita eleii^hts draw on follow-up data after 
case closure to monitor retention of Earnings , by individual' 26 closures, the 
percent of 26 closures who remain non-dependent on publi^c assistance as their 
primary source of^ support, and the percent of non-cqmpetitively employed 26 
closures who rd'tain their enhanced independent living and functional skills.- 
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• The eighth and last performance standard monitors the consumer's 
appraisal of services client satisfaction witK VR services . Two data 
elements include measures of client satisfaction with overall services and 
various asjects of seryices (e.g* counselor promptness, th^ quality of 
placement services). The third data element moves beyond satisfaction to 
monitor ^e client ' s ^^udgement that services received were useful in 
obtaining his/her job or homemaking situation. 

Abandoned in the proposed st^jandards revision are those elements in • 
the existing' standards which focused on post -employment services, manageable- 
sized caseloads, the reasons for unsuccessful rehabilitation, and the length 

. in time of the service process. The proposed new performance standards ♦ 
monitor outcomes and cost-effectiveness, not service proems i 

*In addition to the performance standards', the revised proposed standards 

, include five procedural standards that do focus attention on ^critical pro- 
cess areas and on data validity. "Assessment of 'performance on these standards 
i-s to occur using. instrumentation and procedures (modifications of the'" Case 
Review Schedule developed by the San Diego State RCEP.JX) developed for 
gathering uniform data- from ^ state agencies. The procedural standards focus 
on the validity and completeness of R-300 data, the need for eligibility 
decisions to be based On adequate diagnostic data and to conform to federal 

law^ and regulations, the desirability that eligibility decisions and move- 
ment through, the. VR process be completed in a timely mariner appropriate to 
the n^eds of clients, compliance with the requirement for the Individualized 
Written Rehabilitation Program, and the need for realistic goal-setting , for 
' clients and adherence to the policy of seeking competitive employment out- 
comes when feasible. : . . 

PURPOSE OF THE REVI^SBD PROGRAM STANDARDS SYSTEM ' , ' ^ 

, ~ Three principal purposes .underline all*of the developmental Work 
BPA has conducted on the Standards System. Simply put, the primary* 
purposes of the program standards are: ^ 

• to guide the behavior of state Vl^ agencies towards greater 
achievement;. 
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• to make available information on the state VR agencies 
achievements with respeiet to the goals and functioits of the 
VR system, as measured by the standards data elements; and, 

• to identify possible probleijis and corrective actions, when- 
ever state VR agencies are unable to reach their objectives 
tor achievement* / 

Guiding the> Behavior of the State VR Agencies . 

One unique feature of the revisers tandards system is that it is 
oriented to guiding ahd changing the behavior of' the state VR agencies in 
new directions, not just reporting on past .behavioi^, \Vhereas the current 
standards system calculates perfownance norms based upon central tendency 
measures for the nation as a whole, the revised standards system is 
designed to allow seating ^f future i5erformance goals , based upon the 
individual state's past performance. In the revised system, each state 
(1) sets its 6\m objectives under the standards for evaluativp comparison; 
and (2) has the option of deciding which other state programs, if 'any, 
should provide appropriate comparison for assessing the state 's performance 
,State agencies can consider such things as their past program performance, 
their available resources, the demand on these resou^^es, and their 
particular policies when setting their performance goals. 

In short, t-he revised standards system replaces- a federally-directive 
set of "after the fact" norms with future-oriented goals set by the indi- 
vidial state programs. By setting goals in advance, the VR system can be 
guided in the directions dictated by the states and RSA. However, it 
should be noted that no sanctions are built into the* Program Standards 
system. That is, no punitive actions ar.e tied to the failure of a state 
VR agency to meet its pbjectives. Funding decisions are also not ba^ed 
upon the achievement of certain objectives. Ins.tead,* the revised standards 
system is concerned with flagging problematic attainment, investigating 
possible problems, and identifying and taking corrective actions as 
necessary. . 
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Providing Information ' .... 

The revised standards system shares with the existing standards system 
the purpose of providing information to RSA, to the state VR agenci'^s, and 
to other interested parties, such as 0MB and Congress, on the achievement of 
state VR agencies. Infdrmaftion will be provided on the VR program as sl 
whole, anjl on each state VR agency. Information will be provided on 
current achievement, as well as past achievement. Moregrver, other informa- 
tion relevant to the VR program will be provided as part of the revised 
standards system. • • ^ 

< * 

Identification, of Problems and of Corrective Action 

Another unique feature of the revised standards system is that 
it does not stop when a state VR agency does not meet its objective on a 
particular standard data element. Instead, a newly-developed data-based 
decision suj)port system identifies possible problems and corrective actions. 
This system is designed to efiable program managers to quickly identify whether 
possible problems can be identified or whether further investigative research 
IS required. 

Summary ^ * ' ' 

. In sum', the > focus of the new standards system is state agency manage- 
•ment improvement and evaluation capacity. The federal role is proposed as 
one of necessary data provision, the generation j^nd making available of 
comparison data as appropriate, and the provision of technical assistance 
to the state agency for interpreting standards' data and identifying how to 
improve program performance.- The leadership role .in improving state per- 
formance is assigned to the individual state agency under the revised 
standards system, . , - 
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STRUCTURE OF THE PROGRAM STAND; 




IS SYSTEM 



The Program Standards system has several components, as shown i 
Figure 1: 

• Standards and Data Elements ♦ A set of eight Performance 
Standards and five Procedural Standards, with associated 
data elements/, measures the gpalf and functions of the VR 
program with respect to coverage, cost-effectiveness, impact 
of client services, compliance, data quality, and the proces 
of service delivery* 

• Process for Setting Performance Objectives ♦ A process for 
setting objectives for each state VR agency on .each of. the 
standards data elements provides clear expectations for ^ . 
achievement, expectations that are set in conjunction with 
each agency. 

• Reporting System . A reporting system presents the levels 
of achievement of state VR agencies on the measures of the 
goals and functions of the VR system which are captured in 
the standards data elements. The system also identifies 
those state.VR agencies with difficulties in achieving 
their performance expectations. Background information 

on past achievement, the achievement of other state VR 
V agencies, the components of the data elements, and on 

information data elements are also presented. 
'# Data-Based Decision-Support System . Possible reasons for 
problematic attainment of a'^particular state VR agency on 
a particular data element* ar-e identified, either through 
investigation by program managers or through further 
evaluation research. In addition, corrective actionS' 
are identified for each possible problem. 

As can be seen from Figure 1, all four of these components ate 
oriented to the management of the VR program. 
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I^igure 1 



The Program Standards System 
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Viewed another way, these components work together in a circular fashion 
to insure that the information generated through the careful application of 
the standards by program managers is retained within the vocational^ rehabili- 
tation system for the benefit of future clients* As illustrated in Figure 2, 
the cycle begins with the agency identifying specific objectives for the 
system and then developing reliable and valid measures for addressing these 
objectives. Next, the agency and individual program managers work together . 
in determining the performance goals for each measure.- Once the system has 
been designed and the method for determining success established, the system 
is then rea^y to begin operating. Monitoring of the system takes place through 
the careful implementation of all data collection strategies and through the 
regut^r reporting of this data to the funjiing agency. 'Having gathered all of 
the required data, the agency can then begin assessing the extent to which 
the program's goals were achieved and 'identifying the reasons behind the 
program's inability to achieve certain goals. As ^ result of this analysis 
and data review, certain changes in the program's policies, procedures, or 
components may be made to improve the program's overall performance. Such 
changes are reflected in "t^ agency' s setting of new'perfonhance gbals and 
the establishment of new operating procedures. . . 

The following discussion explores each of these stagers in greater* detail 
and offers specific examples regarding how the revised Program Standards 
System adheres to this model. ' : ' 

I 

Jdehtify Objectives and Measures 

As previously discussed; the objectives .of the standards can, be 
summarized into four Concepts: ' " j 

• • Coverage: Is the agency^ adequately addressing the scope and 
type of needs of its eligible target populations? 

• Efficiency: Is the agency sufficiently productivp> given 
the resources available io it? 

• Impact: Dbi^^We agency help to improve the "quality of life 
of the individual clients it services? Ddes the agency return 
more benefits to society (in terms of wages, taxes, and other 
benefits) than the societal costs* it incurs (e.g. tax revenues 

• expended)? ^ ' . 

. . . . ^ - 33 i 



Figure 2 
Operating Model for the 
Revised Program Standards System 
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Goals and Plan 
for Operations 



Program Responses 
#. changes in policy 

• changes in procedures 

• changes in program 
components 
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• Compliance: Are elibility decisions made in accordance with 
the laws and regulations? Are all of the regulations being 
adequately addressed? . ' 

Each of the 13 Program Standard^ are designed to address one or more of 
these four broad objectives. In some instances, improvement in one of 

^^^ese areas may come at the expenses of another area. For example, a 
program may decide to provide services to more clients (i.e. increase 
coverage), ^nd therefore require greater resources,\a decision which 
may result in the program being less cost-effective. Similarly, efforts 
to improve impacts may fesult in an agency spending more resources on each 
client and consequently, reducing its ^^^iciently level or reducing its 
ability to serve as many clients^ (i»e., reduce coverage). These trade- 
offs occur continuously throughout the life of a- program and are influenc 
by a host of political and service considerations. IVhile these trade-offs 
may result in various objectives being weighted more heavily than others 
at a given point in time, such, trade-offs do not alter the fact that these 
four objectives comprise the basic foundation of the standards system. 

It is one thing to express a program goal such as "increased V 
'economic independence for clients'* or '*use of resources in a cost-effective 
manner." It is another thing to specify the measures for such concepts. 
Criticism of the earlier standards make it apparent that it was the measure 
rather than the concept' of standards or the standards tHemselves which were 

• found lacking. 

To identify the most appropriate data elements for the standards, BPA 
first reviewed the availabilility of data at the state and federal levels. 
The VR system has an extensive client -based data system, based upon stage 
agencies sending data on closed cases to RSA annually, in addition to a 
number of reports and plans containing aggregate data. To pretest alternative 
measures, BPA used annual data tapes and other relevant sources to determine 
which of the possible measures best expressed the intent of the standard, ^ 
which were most readily constructed from existing data systems, and which 
would be of most use to program evaluators and administrators. 
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As an example of the measurement problem and the direction taken for 
resolution, consider^the first^ performance standard, which relates to 
coverage : ' ■ ' • • 

"VR SHALL SERVE MAXIMUM PROPORTION OF THE POTENTIALLY 
ELIGIBLE TARGET POPULATION, SUBJECT TO THE LEVEL OF FEDERAL 
PROGRAM FUNDING AND PRIORITIES AMONG CLIENTS." 

A serious methodological problem that of estimating those Vpoten- 
tially eligible" for service impedes the precise measurement of perfor- 
mance on this standard. No regularly collected population survey Indicates 
the number of individuals with a handicap who has vocational potential nor 
is it possible to derive these estimates through cross-tabulation or other 
manipulation of existing surveys. 

In spite of the lack of a precise measure for the target population, 
coverage* is an important aspect of performance. Therefore, in the absence 
of a precise estimate, a coverage "proxy" was identified. Clients served 
per 100,000 state population. \Vhile this measure assumes an equal proportion 
of disabled across all states (an unlikely situation), it has been broadly 
used by state and RSA as a measure of coverage. \hus it has practical 
utility for VR managers, currently. Moreover, if unable ^estimates of the 
target population were to become available later^ the "3iStf^elements or 
measures for the standard could be refined or long as 

the program mission and values remained the samei '."tli^^^^^dards would remain. 
However, changes in program knowledge or in data availability or experience 
with use of the standards might result in changes \n the data elements* 
or even additions to the standards themselves, as measurement problems are 
resolvedW • 

» 

Set Performance Goals and Operating Procedures 

A major shift in the proposed standards system is for state agencies 
to se^ their own objectives, in terms of levels of expected performance by 
which the state program is to be monitored and "held accountable." The 
existing standards dra/upon central tendency statistics to judge whether 
a state program performed adequately in the past year.. The central tendency 
statistical approach, while descriptive, did not examiMe the level of typical 



performance with what was reasonable, or desirable, but instead automafically 
generated '^failures" and ''successes*' among state programs. The more similar 
stat^. agencies, the more arbitrary the central tendency approach becogies. 
Finally, because the centjal tendency approach required the data for all 
s'tate programs to be available so that the distribution could be calculated, 
performance ''norms*' for state programs were dependftait upon the timeliness 
of state submissions of data. , • * ^ 

The new system for setting performance objectives places responsibility 
within each state to set its own objectives for the level of performance 
to be •achieved in an upcoming fiscal year, rather than continuing with the 
post-hoc system based upon national norms, \ihen setting performance objec- 
tives, state agencies might be ^ticipated to look at their past performance, 
at the , levels of performance being achieved by other state programs that 
agency staff view as comparable, at the performance nationwide, and at pend- 
ing changes in state economic conditions, policies on client and service mix, 
and other unique state factors which might affect performance. RSA may 
provide technical assistance to the state agency in identifying appropriate 
levels, and {Participate in the state's setting of its goals, but the lead and 
principal responsibility in setting objectives for performance for the coming 
Fiscal year would be with the state agency. The new system recognizes, that 
state agencies best understand the needs of their programs, that there are 
appropriate differences among state agencies in policy priorities, and that 
it is the state agency which must accept that there ai'e performance problems 

.or shortfalls if needed improvements are to be .identified and implemented. 
A particular advantage of this relianpd on state agencies to set 
performance level objectives is that it permit! the standards system to 

,be used for monitoring and assessing the ongoing program. State agencies 
can use their in-house data systems to monitor individual data items 
on a monthly or quarterly basis, and to see if thfe program is on target 
in terms of moving toward annual goals or sustaining acceptable rates of 
quality closures. Thus, the standards ^^luation system can provide much 
moro immediate feedback to program management to lead to improvements in 
performance. 



Eventually, state agencies or RSA will^t performance levels that 
would indicate "poor'' performance* In some cases, this would involve a lev^l- 
setting process- that is informed by, but not set by, statistical norms. For 
example, some data elements might best be set as policy levels, not statis- 
tically; it is conceivable that all states could be performing poorly or 
adequately on a given- data element, and that the cue that triggers examina- 
tion of the problem should not merely result in some subset of states being 
identified as having a problem if all states have problem?. Data elements 
recommended for performance levels being set by policy makers as opposed 
to being determined by data include; lii (percent severely disabled) 
4ii (comparison of mean weekly earnings); 3i (percent .closures that are 26>; . 
and benefit-cost. Review of past statistics will help in goal-setting. 
However, BPA strongly urges that the current post hoc statistical norms 
system not be used by state agencies as the basis ^or flagging problems. 
Rather, performance levels should be set to reflect policy goals and be based 
upon reasonable expectations in light of the state's past performance. Trade- 
offs between coverage, impact, and efficiency should be explicitly considered 
in setting state agency goals. IVhile states may ^use the approaclj to central 
tendency for descriptive information dnd comparison purposes, such statistical 
procedures should not set the performance levels for the standards, or be 
the sold basis for state investigation. 

Implement the System and Begin Data Collection Efforts 

In order to acquire the data necessary to determine the extent to which 
each state is meeting its performance objectives, uniform data collection 
procedures must be developed and implemented. As previously discussed, 
one of the criteria used in determining the data elements for each o£ the 
13 standards was the current availability of the data at both the state 
and federal; level. Building upon the existing R-300 system, BPA developed 
a number of additional data collection instruments to Complement the range 
of information currently available to RSA through the R-300. The following 
discussion briefly summarizes the data collection procedures for both the 
Performance and Procedural Standards. 
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Performanqe Standards 

The Performance Standards are designed *to be calculated each fiscal 
year. States already have been routinely 'Collecting much of the data 
required by the Performance Standards. Of the seven separate data sources 
used for the Performance Standards, three are in reports that have been pre- 
pared for RSA historically; or on new report designs: 

• the RSA-300 Case Service Report (providing data on individual 
client outcomes); ^ 

• the RSA-2 Annual Report for Vocational Rehabilitation "(provid- 
ing data on aggregate agency expenditures); and ^ 

• the RSA-113 Quarterly Cumulative Caseload/Expenditure Report 
(providing^^^data on the agency's caseload flow).* 

RSA is currently involved in efforts to revise the RSA-300 and RSA-113 
reports, in response to 0MB requirements. Because of this, the reader should 
be aware that the references to* specific data items may not corresp<5nd to ^ 
specific forms designs now underway. \ 

The RSA-300 report woufd need a few additional data items to re^ond 
to all the Performance Standards. It has four parts which are completed 
at different points in the rehabilitation process: at first referral, at , 
completioa of the referral process, at completion of the IWRP, and a.t 
closure. The information gathered pertains to the clients ' Work status, 
disability, primary source of support, the results of their movement through 
th[e VR system, and other demographic and personal informaiton. 

The RSA-2 has been discontinued by- RSA; the report's information was 
'icluded as part of the proposed RSA-113. However, references to the RSA-1 
have been retained in order to show data collectors the type of ^nfoinftation 
^.^.^^ , required. The precise location of the data (i.e., t|^ report containing the 
needed information) is irrelevant, as /long as the data is accessible from 
somqKvhere within the state Agency accDunl;^ing system. ^ j 

The RSA- 11 5 is a new report created by RSA to gather <iuarterly Infor- 
mation about client flow within<»each VR agency. It shows how many clients the, 
agency accepted in the previous quarter, how many closures were made during 
the previous quarter, and the types of closures. As well, the report pro- 
vides information on the number of applicants and entrants in extended 
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evaluation; gives projections on new atfceptances and rehabilitations; and 



provides information on expenditures* However, as noted above, the expendi- 
ture information is insufficiently^ detailed fpr some data elements ♦ ^ ' 

In addition to the three program reports used for the Performance 
Standards, the s'fandards also call for implementation of two different client 



survey^S: 

• the^Client Closure Survey (providing information on client 
satisfaction with VR services) ; and . / * 

• the Client Follow-up Survey (providing information on client 
retention of benefits). 

These twp surveys are administered as mafl/back surveys, completed by a 
sample subset of the agency's total group of" closed clients for the given 
fiscal year. The Closure Survey functions as the data source for measuring 
a client's satisfaction with various aspects of his/her VR services and 
should be administered as soon as possible after closure from *VR. In con- 
trast, the. Follow-up Survey is used to measure clients* success in maintain- * 
ing, over time, the ''benefits*' resulting from VR service: thus it is concerned 
with whether or not rehabilitated clients have retained their jobs, eamii|jfs 
levels, freedom from public assistance, ai^j^ functional abilities. The 
Follow-up Survey is sent to the client one-yea*r after closure from VR. 
^ Finally, implementation of the Performance Standards will recfuire 
accessing two "exogenous" data sources: 

.# the^ annual b.S. Census publication Statistical Abstract of the 

U.S.. (to provide data on jthe cun;ent federal minimum wage and 

on state wage norms); and 
• the U.S. Bureau of the Census Current Population Reports , 

Series P-2S (to provide state, population estimates). 

Any stjate may prepare the sftandards data items from state data. 
If RSA were to pr-ep^re^the item, states would submit the -jiecessary data to 
, RSA. 
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Procedural Standard's 



The Procedvjipal Standards will be reported for given state agency, 
every third* fiscal year. RSA will conduct? the data collection and will 



report the results to each state'' agency . The> data elements for the Pro- 
ceJur'aT "Standards consist of a number of individual information items 
pertaining to, various aspects of the particular issues addressed by a 
^iven Procedural Jtandards^ Thus, RSA.and state agency program managers 
will be presented with information on "how things are done" in the agency 

* With respect to the key^ processes emb9di^d in/fhe Procedural Standards. 
It is intended for states to use the Procedural Standards to benefit thei 
program evaluation efforts and facilitate the improvement of services to 
clients. Tlhe information obtained via the Procedural Standards will form 

/^tihe basis feg^gency decisional fo^joaake appropriate changes in practices, 
where current processes are n^dxTTiT keeping with client interests and posi 
program perrormance; . • ' , 

The methodology for implementing the .Procedural Standards reflects 
the desire to allow maximum flexibiliJEy to states in the VR process, yet 
\still ensure attention to the/foeas addressed by the Procedural Standards 

^"^d provide sufficient data in these areas to. allow for program-wide, aitjfl 
sis. Ideally, a uniform procedure would be followed by all states for 
monitoring these process areas, even though states retain differences in 
the ways they organize and conduct case service delivery. Indicators of 
•cong5liance-with legal requirements, ?uch as eligibility and' IWRP, should 
be the same for all states; that is, the same questions should be asked 
and the same summary data should be reported. 

Most of the needs of the Procedural Standards^ are b^st met through 
case reviewl Thus, a single case review process will be implemented to 
address the case review needs of *four of the Procedural Standards. This 
process will use thp Case Review Schedule (CRS) , developed by the San 
Diego State RCEP IX, *s the basic document for ProcedCir^l Standards data 
collection. The CRS has already been mandated by RSA as the standardized 
instrument to be used by* regional RSA offices whenever they conduct case 
reviews. For Procedural Standards 10 (eligibility) and 12 (IWRP), the 
CRS items essential to adequately assess compliance have been selected. 
These items make up the Modified Case Review Schedule (MCRS) , which is 
^considerably shorter ^han the full CRS. RSA could choose either the CRS 
or the MCRS as the instrument for collecting Procedural Standards data. 



While the CRS is an appropriate vehicle for collecting cpmpliance 
data, it lacks certain- items needed to assess the validity of R-300 data 
(Standard 9) ,or to assess timeliness of case service (Standard 11) ♦ For 
these standards, two separate instruments have been developed to complement 
the CRS. These two instruments have been incorporated directly into the 
MCRS to provide a unified data collection instruin^t. 

Finally, Standard 13, on the correspondence between the IWRP occupa- 
tional goals and final Outcomes, uses data from the R-300 and consequently 
can be reported annually. * ^ 

Data Reporting . ^ 

The standards reporting System brings together the various sources 
of standards input data so that a particular agency's attainment for a 
specific time, period can be compared to its objectives for the period. In 
addition, the reporting system will provide the program managers with the ^ 
capability to flag and investigate problematic attainment. To do these 
two tilings, the reporting system has been designed 

• to keep track of past performance as well as current expectations; 

• to present the findings in an. ,easy to use, easy to understand way, 
without unwieldy reports, emphasizing graphical presentations us 
well as plain numbers; and 

f to -make sure that the reporting of results occurs in a timely 
fashion, so that future performance can be influenced. 
The standards system is compatible with the kinds of data compilations 
routinely generated even now in many state agencies' internal informa- 
tion systems. Thus, the evaluation standards System could *be adapted by 
individual state agencies for their use; the calculation of national 
norms will require a natiojial data system. 

Tables 2, 3, and 4 illustrate the main reports in the Performance . 
Standards system. The first set of repoFts (dr\e state's example is seen 
in Table 2) will show achievement on each, of the standards^for a given 
agency. In addition to showing this year's performance, the table also will 
show state's goal for the year, its last year's performance, and the 
previous year's national norm. With this information agencies can see how 
successful they were in meeting their goals, for each of the data elements. , 







A-27 

* « 




1 






t 




■ • ■ Table 2 




* 


, f 








^ ' ..... ^ 

ACHIv^VuricNr ON PtftFUKrtANCH: STANfjARDS 








* 


;Y£Ak: l9S'. 
STAlhl: CA»..lFuRNIA 






• 










This 
YtiAK 


THIS 
Y!£A»ib 
GGAl 


.1980 


.'980 » 

NATL 

nORH 


















u J, Xw VI Cf se^rveu utfr v;u,vw 
ootn.iAaT/a on 




* XXX -X 


V V V V 
XXX • A 


AAA • A 






t ? r c =r 1 r •/ tt V T* e J. u ax:- a >3 ie g 


y X X 


XX « XX 


' XX -XX 


XX. Xi' 




COST. 


KbTURN 


• 












•£ X T « n a X ^ i.i r h ' ■ e r c o lo Tfi e % - 


%>XX^/ XX 


'pXX . XXX 


'i>A A 9 A XX 


*XX,XXX 








^XXt^XXX 


''XX, XXX 


'>XX tf Xaa 


->aA V AA A ^ 


■ 




» < a c X o o + c o H/ a 1 -J K: J.» k n e t i s 


xx«/:x 


XX « XX 


A A M A 


AA . AA 






r f ) T/ a J. e '*/ n e n t i "0 + r t) to V Ti* 






aXXa.X 


A XX A M A 


O » 






- 












r'er ceno J.o> c;oir»ures 


XX* A/ 


A A •iV\/» 


AA M A/r 


YV Y 7 ' 
A A . A.<t 




(ii) 


Annua l caanqt^ in numnt^r 
of .6 c j.osur t^'H 


xxxx 


xxxx 


XXXX 


XXXX 




. J... 














«*'''er(:fc:n*:. Jo> r. .'.OHure^ v;xTrrx * 
















XX. XX . 


XX. xx . 


A A « A /. 


Y Y Y 7'* 
AA . A^ 




MX> . 


»..()mi.i»jir ii.^io)! ot Loarnxno v o*- 












i 


c J. oviLvr t*<> "vo v?ar nxn»j i . Dr 

JL^ lU t:» A ( i 1 1 t* <. •> X 1*1 ^/ ■•.*/ a 'tr t> , 




. X.XX ' 


X.XX 


X.XX • 




1 ; i) I. ♦•/•*• 


. nO Va V IT ' 












(ii; 


K e r c e n c: oj^i t:m:> i ^ i v e ]i u 


■ XX-aX 


XX. XX 


XX. aX 


XX. X ' 






aoiiriu earnxnai-> a.tr/aaove 

•^eaera.L mxvixioum wdqe 

rercenv, n()ncc)moeT/i^iveiL^ 


XX • XX 


XX • XX 


xx.><x . 


XX. X.? 


lERLC 




imorovttwenTr in tuncT/ionxnq 
ana ixte i>^aiuH 


XX »»XX 


XX m X/» 


XX ^ X/f 


. XX. Xi* ; 



A-28 



Table 2 



i 

SThTi-..: CAL-fFORNlH 



. .THIS 



THIS 
GOAL 



L980 



f^ATt. 



O^lisf'i' CHANGE 

(i> CoinT:iari.Hon of earnmas 
ana t anc^xonxnci oDiii x^u 



•XXXx . XX 
XXX, X 



',xi; J^ompar I'.ion of 26 cio'iure*?. 
wicn T»u.Oi.jLC assistance as 
riPimaru souxct^ of' <>uo'Cior";, 
o*t ^:.i(;i<-*^ur^ one at tc/i. jc»w-un 

• 111; >-».>rcTuMi nonr oniDi:;^ :i.ve.;.Cf 

o ^ j. o w-u 0 . ^ X/i 

•^'erCfc^nT, (:;'.n^>eG clients 
, satistiea ov^^rai!. v^'^ 

e X ti r 1 e n c w • X X • / 

"1 . . ^ . . 

satisfied with: 

counije.'.ur . XX -XX 

p A u '5 i c I /. J. ' i> <;> t o V a t .l a n X X « X*^: 

T.va:i.n:j.na i^er v.i.ct*s XX - X/.' 

* • Dia^:t;jm«n ?>er v.ixft.-: . XX«Xi' 

rM;r..,».i « ii.nKj joo/ 
a o M) I asc r St. m.^ c x o n o ( • i 
•:ar t-^-jH". T^.r f Onnanctt XX-X/i 



'XXXX.XX 
XXX Ix 

^ XX 
XX -XX 

XX iT A/'" 



XX » X// 

XX . x;i- 

XX. XX 
Xa n X/m 



XX . XX 



XXX A m Xa 
"XXX.X 



xx . x;r 



XX -XX 



XX .4/ 



aX i^x/ 

V 

XX - A. A' 

A A « Xyw' 
XX - A/ 
XX M X/i 

XX. XX 



CXXX . XX I 
XXX. xl 

XX -Xi' i 

xx.xx 

XX -X/ 



XX. X/ 

xx*« a/ 

X'X.x;-." 

XX • X/» 

X X • X z^'. 

XX.,Xi'i 




45 



A.29 



They can also compare this year/s performance with last year*s t-a see where 
they have and bave not improved* Finally/ agencies *can assess their current * 
performance in relation to recent national norms. This type of report 
gives prograrajnanagers an overall view of agency performance, while at the 
same time pointing out specific strengths and weaknesses, currently and 
over time. A particular ^advantage of such reports is that their "turnaround" 
time can be relatively short. The short turnaround time is possible because 
tlie reports use only the individual agency^s data (and a previous, year ' s 
national norm). Computing the current year's national or regional norms 
requires data submissions from all relevant states. .Thus, production time 
for reports like that shown in Table 2 will be a function primarily of the 
agency's own data preparation. 

In addition, reports could be prepared for each data element which" will 
display all agencies performance. on each particular element. Table 3 shows 
an example for data element l(i). This year's goal as well as performance 
in the four previous years will be present ed^^^^^^^encies can use the informa- 
tion to compare their performance and thprtoals to other similar agenciesT. 
By providing data for the four previous years, trends over time can be 
analyzed. Agencies and RSA will be able to determine if performance has 
steadily improved over time or if this year's performance is noticeably 
different than previous years. 

Finally, Table 4 shows an example report of national performance for 
each- data element for all ag'encies, and for general, combined, and blind 
agencies. This allows a program-wide view of performance in VR. - 

These three, types of reports will be generated routinely for all of 
the agencies and all of the data elements. In addition, RSA and the agencies 
will have thgsj^apability to use the\ystem to generate special purpose 
reports and ^na^lyses. For example, the basic reports could be run separately 
for special population. These may^take the form of statistical reports or 
of graphic displays. ^ ' *' 

Finally, the system will provide access to a large number of supporting 
information items useful in analyzing :the interpreting the routine reports. 
These inlformation items feed in\o the decision support system, discussed 
earlier. Based on any problems Ivhich emerge in the ^ency's standards 
performance, program man^{gers will inspect particular/information items iS^ed 
to thje various standards data elements.'* 
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Table 4, (continued) 
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Performance Assessment and Policy Analysis 

Out of the» standards reporting system will come the clear indication 
that some agencies will pot haye met their objectives for level. of attainment 
on some data elements. The standards system, doeg not .stop there, however, 
but instead provides a system for investigating the causes for problematic 
attainment and for developing corrective actions as part of the decision-^ 
suppbrt system. This system is described in detail in the Analytic Paradigm 
for thq^lVR Program Standards /^ but can be illustrated briefly here* Basically, 
the depision^-support^system. is designed td provide VR program managers with 
^^iniovm^ion wKich is: - ^ 

• relevant to the issues (i.e., problems) underj;orfsi 

• quickly and easily interpretable; . ' " 

• timel}^; and ^ e 

. • suggestive either of an immediate policy response to the 
problem, or of further investigation 'needed before an 
^ appropriate response can be formulated. 

* The basic flow of the decision-support system is shown in Figure 
3* Problematic attainment, where an "agency is unable »to meet its agreed- 
upon objective for a particular standard data element, is the signal for 
the process to start. First, program managers within RSA and within the 
state VR agencies investigate the problematic attainment. If they are 
able, to identi:^y problems and possible corrective actions^ then imple- 
mentation is the next step. If not, thfen more formal evaluation research 
is called for. ImplementaXion of the correct^ive actions will affect state . 
VR agency opera>tions in the next cycle of the standards system; As a 
resuTt of the cor^rective actions, the -agency may be able to meet its obr 
jectives. Otherwise, the cycle starts anew. 

As noted, the investigation of problematic attainment has been broken 
into two parts: ' ' . 

4 / 

'1 • 

Berkeiipy Planning Associates, Program Standards Evaluation System, 

Final Report Volume JI, Analytic Paradigm for the VR Program Standards . 
This ]:?eport i^ available from RSA. 
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Figure 3 

The Flow of the Decision^Support System 
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• basic problem identification , carried out by .program 
• ' ! * ' * 

managers within RSA and within the state VR agencies, 

using the standards ^reporting system plus the managers' 
knowledge of program operations;', and- 

' » *k 

• evaluation research, carried out by evaluation researchers 

*■ ' — *» 

A , within RSA or within the state VR , agencies, or by outside 

consultants, existing data bases as Well as new data 
collection. ^ - * ^ , « 

These 'two parts differ in who carries them out, but especially to the ex- 
tent that th^ basic problem identification occurs in a timely fashion, 
ing the reporting system and the MIS-* If evaluation research is re-- 
quired, then most likely corrective actions will not be passible in "ti|ne 
for the next cycle of the process. In tact, 'the results of the evaluation 
research may not be ava^able for 'a year or more, given the nature of 
evaluation research. This lag is the reasons that the investigation of 
problematic attainment is .broken into two parts, so that timely corrective 
actions can be taken, if possible* 

The Process of Problem Identification |f' 

The process of problem iclentif i'cation outlined below is to be carried 

out by program managers, within RSA and within state VR agencies. The 

information for the problem identification will come from the standards 

reporting system^ as well as from the managers' knowledge of program ^' 

operations. The process consists of tracing the possible problems by 

first organizing the components of the standard, then examining as 

* 

**second-lever' indicators other data, elements and other informational 
elements of the reporting system. Examination of these will then lead 
to further examination of third-level indicators, and so an. At any 
point in tracing out these indicators the problem may be identified to the 
manager's satisfaction. At that -point, corrective action is formulated-. 
Or, at any point in \^racing out these problems, further analysis in the 
form of evaluation research may be required. This process is like- 



that normally illustrated by a decision tree. Of course, the process of 
problem identification may lead down several paths at once. Also, more '* 
than two paths may need investigation from a particular node, or more 
than three levels of indicators may have to be examined. The point is to 
do the analytical thxnking and utilize existing information to identify 
possible problems and corrective actions/ This process is illustrated in 
Figure 4. ' - 

If a data element shows problematic attainment, t^e first level of 
analysis is to examine the components of the element, dissecting the ratio * 
or measure into its separate parts, to pinpoint the areas needing attention. 
For example, if the numerical Value of a ratio. is too large, the problem * 
may be in the numerator (too large), th^e denominator (too small), or both. 
Comparison of attainment on the data elements or their components with that 
of other agencies with s^fmil^ar programs, or historically, or on. other data 
items, can help determine the extent to which "the .indicator shows a real 
problem or if there is a good explanation for the attainment/ The goal in ' 
this analysis is to seek explanation , or the identification of which 
components or related njeasures pinpoint the areas to be explored^ further. 
This analytical process may take several iterations bfefore a cause is 
pinpointed. The first levels of^^e-*pjgcices^s are ipt ,to be seen as complex 
statistical analysis problems, but rather straightforward, simple' program 
comparisons that allow VR managers to progress through a dedBsion tree, I 
diagnosing problems and using ^program inforni^tion|tfP^aaqlifi iBoAclusions 
about probable causes. 'Some branches "of a decision tree process may le^d 
to* problems or investigations that require complex statistical analyses, 
but only after several' levels of^he process have occurred. 

Table 5 shows the decision steps in an example exploration; this 4s 
a model ^tor investigating the possible causes or problems if '"'expenditures 
per 26 closure'*, data^ element 2(ii),is problematic/ \ . 



^Thc 'Analytic Paradigm projifcieS similar.^ decision trees for other 
Performance Standards data elemWts. 



\ 
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Figure 4 
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Table 5. 



Investigating Inadc<.jate Perfonnance on Data Element 2fiil ; 
Expenditures rer 26 Closure 



Seen 
ario 



5S 




Impli- 
cations of 
Pi^st Level Indicators [ First Level 
Cost/Closure]~Co-;t/Case I Indicators 



Acceptable 



Un- 
acceptable 



Acceptable 



Acceptable 



Agency is 
^ichieving 
too low a 
proportion 
of 26 clo- 
sures 



Second Level 
Indicators 



"Leading 
I Questions" 
(and answer] 



Standards Data 
r.leroent 3(i) 

126 \ 



Is the % too 
low? 
If yes, why? 



Acceptable 



Agency is 
serving 
clients 
too 

slowly; 
adiieving 
too few 
closur^ 



26*l28*/»307 



If no, which 
clients or 

I components 

! cost too 
nwch? 



Un- ' j Agency has 

occeptable^l recently 
developed 
a bottle- 
I neck in 
intake 
I process; 
too few 
' clients • 
I being 
accepted 
into the 
system 



MIS element; 
Post-Acceptance 
Closure 
Rat^ . 

^>26+#28^Jf3a\ 
open cases/ 



1. Id the 
service , 
I process 
too slow? 



2. Have we 
had a recent 
influx of 
acceptances? 



1 . Standards Data 
Element l(ii), 
served (10-301 



^100,000 population 

2. MIS element; 
Rate of Acceptance 



1. Do we have 
too few 
applicants? 



It of new status lOsN 



9 new applicants * ] 
* on-band applicants^ 
<f on-hand 06s / 



2, Does use 
of Hxtended 
Evaluation 
account for 
the low 
acceptance 
rate? 



J* Do we have 
too many in- 
eligible 
applicants? 



Third Level 
Indicators 
(if applicable] 



None (go to next 
column) 



• Administration 

cos^ts 



• Service costs 
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— 28s and 30s 
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(frqro^thc Mis;i 



• Service costs 
by service 
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rate of 
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of applicants 
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too long) 



MIS element; 
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Research Questions 



Conduct Outcomes Analysis 



1. What proportion of total costs go to ad- 
ministration? 



2. What .is the average life-of-case cost for 
each closure group? 

3. What proportion of total life-of-casfe 
costs are spent on each closure group? 



1. What proportion ©^ current, service costs, 
went to each service type? 

2. What is the average cost of each service 
^yv^» for clients receiving that service? 



Which aspect of services for accepted clients 
takes relatively too long? 



None (end of investigation) 



Could outreach be made more effective? 



. What kinds of clients are going into 06? 

. What kinds of services are provided during 
06? . 



1. What T6asons are given for closing , clients 
ineligible? * 
^. From where are these clients beipg 
referred? . - ' 



> 
I 
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The column headed "first level indicators'' shows four possible com- 
binations of two other indicators, cost/closure and cost/case, which are 
used to investigate an unacceptable (high) value of data element 2(ii). 
Depending on acceptable or unacceptable levels of these indicators, a 
different ''scenario.,*' or type of problem, is identified. For instance, 
if both of these indicators are "acceptable," then this indicates that 
the agency is achieving a proportion of 26 closures which is too low. 
This can be confirmed by referring to data element 3(i). If cost/closure 
is unacceDtable, but the cost/case is acceptable, them the agency is achiev 
ing too few closures. As can be seen here, this first-level diagnosis 
leads to in-depth investigation of different parts of the system. The 
table shows the! types of second- and third-level questions that could be 
pursued, depending on the iriitial comparisons and explanation. 

At each level of the investigation, the goal should be to quickly a^id 
more finely tiJie in 'on the precise nature (i.e., cause) of the problem. 
Depending on the findings generated by a given level of the analysis, the 
program. manager ^ay decide either; • that further inveWigation is warranted 
before formulating a policy^^response; that the findings ^are adequate to 
suggest art appropriate* response; or that, despite- the adequacy of the 
findings, no useful policy response can be offered (e.g., due to prior 
institutional, legislative, or funding constraints) , 



% The indicators used, in 'the .investigation of problematic attainment 
are grouped and sequenced in' such, a way as to answer increasingly detaile^ 
questions. This allows ^anagers to go a fair distance in determining the 
nature of the problem before needing recourse to'mojre. sophisticated and 
time-consuming "causal" analyses. This, is not to say that other sophis- 
ticated analyses are undesirable o^ urtntecessary. On the contrary, they 
as often as not may^ prove useful to managers in pinpointing precise causes 
of problem performance. However,' the ^^dvantage of' this model is that it 
allows managers to quickly investigate\and discard certain hypothesis re- 
garding the problem's cause, ahd thei^ef0re to more quickly direct' the 
investigati,on toward what seems, to be the likely cause. Once the lilceiy ' 
cause is identified through use of the /indicators , the manager can direct 
the ovaluation/i:;gsearch staff to- conduct the needed causkl. analysis . 



Evaluation Research 



As noted above, corrective actions may not always -result jErom the 
problem identification procedure. Instead, the program manager may need 
to conduct "causal" evaluation research and pro gr^ 'analyses To^ddterai^ie 
the source of program performance problems. These (often multivariate) 
analyses control for various state factors which simultaneously influence 
performance. Such research often examines the VR program as an inter- ' 
related system of activities and may require special data collection. - 

Program Response. 

Y The key to effectively using the standards system as a management tool 
|ill rest with the ability of[jRSA and the states to incorporate new proce- 
dures or policies which may emerge from the careful analysis and review 
of the standards' data. These changes may involve policy decisions; 
federal and state congressional legislation and regulates; resource 
committment adjustments; data system revisions; technical assistance to thev 
states; research agendas; university counselor tr^namg programs; and 
coordination with other programs. The primary actors with responsibility * 
for making changes in the standards system are the same "as in the VR 
system at large: Congress, 0MB, Department of Education, RSA, Regional 
Offices of Rehabilitation Services, state governments, and state VR 
agencies. The set of actors and associated types of corrective action 
include: ^ . 

• Congress, 0MB, Department of Education: Funding levels, ' 
allocation formulas, 'priojrities to client groups, procedural 
requirements; 

RSA: Regulations, monitoring, evaluation. Research (along with 

• NIHR), program development, guidance materials, training 
programs, demonstrations; ' * * 

• Regional ORS: -Technical assistance 1;o state VR agencies, 
dissemination of information; diffusion of innovations, 'training; 

• State governments: funding levels; artd 

• State VR Agencies: same as RSA (e.g., regulations, evaluation), 
eligibiUty determination changes, counselor trailing, ease 
management changes) servi'ce provision chan'ges; managemetit of " 
sub-units (e.g., districts, offices). 
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Evaluating th^ Standards System Over Time 

One of the problems with the current standards system is that no 
evaluation of the use of the standards is included.- For the revised 
stanJaWs syste^ such evaluation^is "cleaFIy' includeav ^ — ™ — 

The criteria tor evaluating the revised- standards system are very 
simple. The most importsmt evaluative criterion is whether the attainment 
of the state VR agencies is improving , in the areas measured hy tSfe standards 
data elements. IVhile it may be very difficult to prove that the cause o£ 
the improvement was the implementation of the standards, at least the 
attainment of the agencies after the implementation can be compared to 
their attainment before the implementation. The second evaluation criterion 
is whether the^tate VR agencies are meeting their objectives . If they 
never meet their objectives, then the objective setting process is not 
working properly. If they always meet their objectives, .then the process 
is also not working prdperly. Identifying for which state VR agencies, 
for whic^ data elements, or,* for both in combination, which objectives 
are not being met will indicate where attention needs to be paid in the 
standards system. The third evaluative criterion is whether the program 
managets find the system useful . Program managers should.be regularly 
canvassed for' their recommendations, 

Qianging the Standards System 

A key word for the standards system should be flexibility . As the 
standards system operates, several factors outside the system may change: 

• the goals and functions of the VR program may change, necessitating 
changes in the standards; 

• reporting requirements within or without VR may change, changing 
what will be available for. the reporting system; 

• the actors .and types of corrective actions possibly may change; 

• actions taKen by state VR agencies might piish the *VR program in 
undesirable directions, as state program -manag^ers try to respond 

to the standards system, thus requiring additional standards or * 
changed expectatJ^ions; and 

• the achievement of the .state VR agencies may not be improving 
over time. 



ERIC ' , '^'t B2 



A number of factors inside the^ system may need change: 
• some data ele^nents may be found to have lower data quality than '* 
is acceptable^ and thus require new procedures or even replace- 



ment;- ^ ^ - 

some of the data collection activities may require ^ change, because 
of logistical problems; 

difficulties in the reporting system and in -the reporting cycle 
may arise; and • ^ 

objectives b^ng set may not be correct* 



As such, RSA must monitor the operation of the standards system over 
time. In the beginning, the system should especially be closely monitored, 
so that problems can be discovered early, and RSA mus|f_J^. ready to' change 
the standards system as the. need arises. 
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COMPUTING AND PRESENTING THE 
^ EIGHT PERFORMANCE STANDARDS 



OVERVIEW 



The Performance .Standards consist of eight goal statements for the VR 
program, and data elements to be used in measuring achievement of those 
goals. The Performance Standards focus qn outputs of the VR program: 
that is, on client outcomes and agency productivity. They provide measures 
of an agency's level of coverage of the eligible population, efficiency in 
service provision, and impact on clients' lives. 

States already routinely collect much of the data required by the Per- 
formance Standards. Of the seven separate data sources used .for the Perform- 
ance* Standards, three have been in use or recently designed by RSA: 

• the RSA-300 Case Service Report (providing data on individual 
client outcomes); , 

• the RSA-2 Annual Report for Vocational . Rehabilitation (provid- 
ing data on aggregated agency expenditures); and 

• the RSA-113 Quarterly Cumulative Caseload/Expenditure Report 
(providing data on the agency's caseload flow). , 

The. RSA-300 report has^een expanded to provide certain additional data 
needs required by the Performance Standards. ^*It has four parts which ar^ 
completed at different points in the rehabilit^ation process: at first refer- 
ral, at completion of the referral process, at completion of the IWRP, and at 
closure. The information gathered pertains to th;e clients '^■'^ork status, dis- 
ability, primary source of support,, the results odf* their movement through the 
VR system, and othex demographic and personal infoi:mation. 

The RSA-2 has been discontinued by RSA, and the ^repo'i^ifl's 'information is 
part of the proposed RSA-113.^ However, we retaiti a reference*to the RSA-2 
because the RSA-/rf3~f inancial information is insuf fici'ently detailed for the 
benefit-cost data elements.. Our concern is to show data collectors f the, type 
of information required; thus we include th^ RSA-2 ,to. illustrate the speci- 
fic informatioij needed. The precise location, of the data (i.e., the report' 
containing the needed information) is irrelevant, as long as the data is 
accessibly fr6m somewhere within the state agency ;accounting system. 
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The RSA-113 is a new report created by RSA to gather quarterly inforraa- 
tion about client flow within eaqh VR agency. It shows how many clients the 
agency accepted in the previous quarter, how many closures were made during 

the previous quarter , and the, types of closures. As we 1 the reportLpxQ-_ 

vides information" on the number of applicants and entrants in extended 
evaluation; gives projections on new acceptances and reliabilitations; " and 
provides information on expenditures. However, as noted above, the expendi- 
ture information is insufficiently detailed for some data elements. 

In addition to the three program reports used for the Performance Stan- 
dards, these standards will also require implementation of two , different client 
surveys: ^ 

• the Client Closure Survey (providing information on client • 
satisfaction with VR services); and 

• the Client Follow-up Survey^ (providing information on client 
^ retention of benefits). 

These two surveys are administered as mail-back suryeys, completed by a* 
sample of the age^ncy's total group otiTTosed clients for a given fiscal year. 
The Closure Survey functions as the data source for measuring a client's 
satisfaction with various aspects of his/her VR" services. In order to tap 
the person's opinions while the VR experience is still ''fresh in mind," the 
survey must be administered as soon as possible after closure from VR. 
In contrast, the Followrup Survey is us'ed to measure clients' success in 
maintaining, over time, the "benefits" resulting from VR service: thus it 
is concerned with whether or not rehabilitated clients have retained their ^ 
jobs, earnings levels, freedom from public assistance, and functional 
abilities^. The Follow-up Survey is sent to the client one year after closure 
^om VR. ' ^ 

Finally, implementation of the Performance Standards will require access- 
ing two "exogenous" data sources: 

• -the" annual U.S. Ceiusus publication Statistical Abstract of the 
« LLi. (to provide data on tlie current federal minimum wage and 

• on state wage norms); and 

® The U.S. Bureau of tKe Census- Current Population Reports, Series 
2z2^ (to provide state population estimates)-. 

Each. of these data sources will be accessed by RSA, and RSA will input the 
required .data into the MIS for computing of the relevant data elements. 
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INSTRUCTION FOR COMPUT^^ THE DATA ELEMENTS ' ' 

STANDARD 1: VOCATIONAL REHABILITATION SHALL SERVE THE MAXIMUM PROPORTION 
OF THE POTENTIALLY ELIGIBLE TARGET POPULATION, SUBJECT TO THE 
LEVEL OF FEDERAL PROGRAM FUNDING ANP-PR-IORtTI^S^AMONG CLIENTS . . 

Data Elements: (i) Clients served per, 100,000 population 
(ii) Percent of clients severely disabled 

This . standard addresses the extent to which the vocational rehabilita- 
tion program is serving the eligible target population: The need to ensure 
accessibility of services to all the eligible disabled is of paramount import- 
ance to RSA and the states. , 

Data Elemeni^^) : Clients served per 100,000 population 
Rationale 

Although this data element does riot provide a true estimate of the level 
of coverage of the eligible target ^population, it provides an adequate proxy 
, measure of the target population in terms of the total state population. Also, 
the data item is ^currently used by state agencies and, therefore, has manage- 
ment utility and validity as a performance measure. 

Formula 

Annual number of clients 

State population (in lOO/OOO's) 

Data Sources 



• RSA-113 

• U.S. Bureau of the Census, Series P-25 

Data Element l(ii): Percent of clients severely disa.bled 
Rationale 

The proportion of severely disabled within a caselx)ad can reasonably b^ 
expected to impact negatively tpon a state agency/s caseload size and on its 
total costs. IVith a high proportion of sev^^y disabled clients, time in 
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process would be expected l:o increase and counselor capacity decrease, thus 
decreasing a program's overall caseload capacity. To effectively assess • 
coverage, the proportion of the caseload that is severely disabled must be 



"Xakeir into" account . Further, given the legislative importance attached to 
service to severely disabled, it is most appropriate to include this data 
element under the standard on cbverage of the eligible client population 



. Formula 



Annual number of severely disabled clients" served 
Annual number of clients served > 



Data Sources 



• RSA-113 



STANDARD 2: THE VOCATIONAL REHABILITATION PROGRAM SHALL USE RESOURCES IN A 
COST-EFFECTIVE MANNER AND SHOW A POSITIVE RETURN TO SOCIETY OF 
INVESTOENT JtN VOCATIONAL REHABILITATION OF .DISABLED CLIENTS. 



Data Elements: 



(i) Exf>^nditures per^competitively employed 26 closure 

(ii) Expenditure per 26 closure 

(iii) Rati^ of total VR benefits to total VR costs 
(benefit-cost ratio) 

(iv) Total net benefit from VR services -(discounted net 
present value) 

Two issues are addressed by this standard. The first is the. issue' of 
cost-effectiveness: with the financial resources available to \he state, how 
successfully did it achieve ^desired objectives? The second issue revolves 
around cost-benefit concerns: are we getting more out of the program than - 
we put in? 

Data Element 2Ci)*: Expenditure per competitively employed 26 closure 



Rationale 



This data element compares total agency expenditures to the number of* 
competitively employed 26 closures* it applies the most stringent criteria 
Xo the measurement of cost-effectjiveness by focusing on pnly those 26 clo- 
sures who are competitively employed at the time seajj^ices terminate. \Vhile 
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this 'data element closely parallels element 2(ii) (expenditure per 26 closure)^ 
it is included because of the long-standing consensus that competitive employ- 
ment is the highest quality and most desirable type of closure obtainable. 



Formula 



J 



Total agency expenditures 



Number of competitively employed 26 closures 
Data Sources 



RSA-2 
RSA-300 



Data Element 2(ii): Expenditure per 26 closure 
Rationale 

This cost-effectiveness measure -relaxes the measurement criteria, assess- 
ing value to all types of rehabilitations. It recognizes that some clients 
are not capable of achieving competitive employment and that other employment 
outcomes can represent achieve^ment commensurate with these clients' abilities. 
'This data element compares tofal agency expenditures to all 26 closures, thus 
capturing the effect of gainful activity, whether it lies in the realm of 
competitive or noncompetitive employment. 

Formula 

V Total agency expenditures ^ --j^ 
Number of 26 closures 



Data Sources' 

• RSA-2 
RSA-300 
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Data Elements 2(iii) and (iv) : 

(iii) Ratio of total VR benefits to total VR costs (Benefit-Gost ratio) 

(iv) • .Vet total benefit from VR services (Discounted net present value) 

Rationale ' ' 

■ " — " ■ \ ■ 

• Because these two data elements are very similar in concept, they will be 
discussed toget,her. Benefit-cost modeling of social service delivery ^stems 
currently enjoys wide acceptance as a measurement tool, with usage extending 
far beyond the VR field. The figures provided by .benefit-cost analysis yield 
a single number which is an immediate indicator of program success. Unlike 
cost-effectiveness measures, which determine the unit costs for achieving a 
given objective (such as costs per competitive closure), benefit-cost models 
estimate total benefits and total costs in terms of dollars. These models 
are neutral with regard to tv^pe of ^delivery strategy. As such, they do not 
penalize ardencies which choose to spend more per client in order to -produce 
better results. Because of their surface simplicity, and because they are a 
popular sophisticated analytic- tool for evaluating program worth, benefit-cost 
measures of the VR system are included in the Performance Standards. 

As a review for the National Science Foundation has noted, benefitVcost 
applications in the VR field are more extensiv'e and have generally beenMnqre 
sophisticated (or at least at a higher level of technical quality) than in 
most other social service and manpower program areas. ^ There are a number 
of models available for us. In one case, RSA commissioned the development 
of a model for routirie U3e by the program, which was designed to be adaptable 
to the needs of. many users (i.e., state agencies, RSA contracted evaluation 
studies, RSA itself) and to be capable of periodic updating and refinement as 
new data became available. That model, developed at the University 'of Cali- 
^fornia, Berkeley, and subsequently refined biy BPA staff, has been used by RSA, 
several state agencies, the Urban Institute, Abt Associates, National Analysts, 
and Greenleicrh Associates, among others, usually under RSA recommendation. . 



^Bcrkouitz and Anderson, PADEC A,n Evaluation of an Experimental Rehab- 
ilitation Project , Rutgers University, 1974. . ^ 
'I . * 

"Frederick C. Collignon and Richard Dodson, Benefit-Cost Analysis of 
Vocational Rehabilitation Services Provided to Individuals Most Severely 
Ilandicapped (TSMH) , Apri*l 1975. 
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This model is the basis for the two data elements proposed for use in measur- 
ing benefits in relation to costs in terras of: 

(Benefits) 

• a ratio — ' 

. ( Costs ) 

• a net difference (Benefits-Costs) 

Currently, the BPA model does, not account for ^ains in functional ability' 
and life status (although it does include monetary valuations for the unpaid 
output of npn-wage earning rehabilitants) . However, the model is currently 
undergoing revision by a project at the Texas Institute for Rehabilitation 

4 

(TIRR), which will develop subsystems within the model to account for such 
functional and life status gains. Because of these impending revisions, we 
cannot include. the precise mathematical formulation for the model in these 
materials. However, upon final revision the benefit-cost model will be incor- 
porated within the MIS, and the interested reader can obtain documentatioji on 
the mathematical formulation from RSA. ' Further, we can specify the components^ 
of program benefits and program costs which are in the current version of the 
model, and which will remain after final revision. 

Both of the benefit-cost data elements use the discounted present value 

1 * 

of social benefits and costs, and rely upon the same components to arrive at 

benefits and costs. These components are as follows: 
Bene fits 

• discounted value of paid earnings; 

• change in output of homemaker closures; 

• change in output of ubpaid family workers; 



A comparison of the full costs and benefits of a VR program can^be 
undertaken from several perspectives.' Perhaps the most common benefit-cost 
perspectives ar<^ the *'taxpaver*' perspective and the "social" perspective. 
In taxpayer EC, we compare direct administrative and service costs of the 
VR program as well as the costs of other government agencies providing bene- 
fits and services to the client population (SSI, SSDI, Food Stamps, Medicare, 
other employment and supportive services) with benefits such as taxes that 
successful rehabilitants pay from their earnings and savings in public assis- 
tance. Social EC takes the broadest perspective, incorporating the \ii(iest 
range of costs and benefits and including on the cost side, for example, costs 
borne by clients and, on the benefit side, cli^ent earnings as an addition to 
the GNP. - ■ P ^ 
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• change in "after hours work'' C®-g-» homemaking tasks performed 
by wage-'eaming rehabilitahts) ; 

• fringe benefits; 

• change in output of families of rehabijitants (as ^result of 

rehabilitants assuming homemaker tasks); 
reductions in public assistance benefits; and ^ 
repeater costs (a ''negative benefit") . ' ' 

Costs 

• total program costs during the fiscal year, minus carry-over 
costs and maintenance costs; 

• costs borne by parties other than VR; 

• research, training, and demonstration costs; • 

• benefits foregone by clients during participation in VR ser- 
vices (i.e., any wages and fringe benefits foregone by clients 
with earnings at referral); and 

• client-borne costs for VR services. 

The model uses two basic types of input: (1) "variables" which are 
input or computed, from program documents (e.g., the RSA-300 and RSA-113) 
for the year in question; and (2) "parameters" which take the form of con- 
stants which are derived by estimation or inference based on previous related 
research, current macroeconomic conditions, and so forth. Again, we cannot 
include an exhaustive list of all the input variables and parameters which 
will be required by the final revised model. Upon final model revision, the 
interested reader may obtain docuraetitation^on all input variables* and para- 
meters from RSA. ' . 



A few final notes are in order with regar/i to the components of the 
ctfrrent version of the model, as listed above. The costs associated with 
homemakers and .unpaid family workers are the sam.e as those for any other. 26 
closure. The benefits of a homemaker are determined, by estimating the "worth' 
of homemakers in the genera^ population; that is, by estimating the dollar 
value of the various funct$^<)ns performed by a homemaker. The^worth of dis- 
abled homemakers is assumed to be some proportion (less than 1) of the worth 
of homemakers in general. .This proportion is then estimated to be the same 
as the proportionate ,worth|x)f disabled workers to normal workers. Unpaid 



family workers are treated similarly. The value of a sheltered workshop 
employee is his/her market value, i.e., his/her wages, '-s^ard less of whjfether . 
they are above or below the minimum wage. 

There' is a term in the model for workers who have been displaced by^ 
handicapped workers. The term estimates the negative impact on these dis- 
placed workers. The term currently has a value of zero because there is no 
evidence of substantial impact in today's economy. This is, of course, not 
relevant to BEP or sheltered workshop employees. 

The net benefit measure (B-G) is included amorig the standards data ele- 
ments primarily because it is the preferred approach of economists. The 
problem, with the measure is that it is very sensitive to the scale of program 
operation: in the case of VR, for example, larger agencies would produce 
greater total net benefits than small agencies, simply because of their larger 
caseloads. Thus, the measure is inapprc^priate for comparing across state 
agencies, although it is useful for observing change over time within an 
agency. The ratio measure (B/C)' overcomes the problem of agency size, thus 
allowing for comparisons across agencies. As well, B/C can be used to observe 
change over time within a single a'gency. 



Formula 



\ 



2(iii) 



(Benefits) 



( Costs ) 
2(iv) (Benefits - Costs) 



Data Sources 



RSA-300 



RSA-2 

• RSA-113 

• Follow-up Survey 
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STANDARD 3: VR SHALL MAXIMIZE THE NUMBER AND^ PROPQKTlON OF CLIENTS ACCEglEa 
FOR SERVICES WHO ARE SUCCESSFULLY REHABILITATED, SUBJECT TO' TOE 
MEETING OF OTHER STANDARDS* ~ ^ ^. 



"26, 



Data Elements: (i) Percent 26 closures 

I ^ (ii) Annual change in th'e number o£ .26- closures ' 

Traditionally, success in VR has been measured by the number' of 
closures," or successful rehabilitations bb^ained* Because a (fentral goal 
of VR is^to rehabilitate .clients, it is e<|sential that the^^stan^a^uls system 
include' a^ way of. presenting how many individuals wer^ successfully served 
and the extent to which this number increases over time. / 

Data Element 5(i): Percent 26< closures . ' - . 

Rationale .3^ 

This data t?lenent^,.fyj<&vides a straightforward measure of an agency's 
success in rehabilitating th6 clients it accepts for services. The data 
element focuses on the proportion of clients, accepted fqr'servicV, (i.e., ^ 
^excluding 08 'si), who S{fe successfully rehabilitated. ^ ^ \ 

Formula - . 

Number of 26 closures, 



Number of 26 + 28 + 30 closures 



Data Sources 



• RSA- 113 V ' . ■ 

... - ■■• . ■ : • • ■ 

Data Element 5fii): ^Annual change in the number of 26 closures 

\ . , ' ' ■ . ^ 

' Rationale ^ . 

Thi:S^^.d^ta qlement attempts to assess an agency* s succes^ in maximizing 
the numbe^r cUentsi accepted ^^6?^rvices, who are successfully rehabili- 
.tatcd. The measure uses the state agency *s prior performance aS a baseline 

* ' A ^ ^ i 

for determining success 'in'^^maximization . " An agency is judged to h^ve ^maxi- 
mized the number of rejtyabilitants if it h^s increased the number of 26 closures 
by some previously specified' amounx, as set by the state ^agency, cpVijunction 
with RSA. ^ • ^ 



Formula \ 

(Number of 26 closures in current year) 
- (Mtmiber of 26 closures in previous year) 

Data Sources 



• RSA-U3 * . " • 

STANDARD 4: REHABILITATED CLIENTS SHALL EVIDENCE ECONOMIC .INDEPENDENCE 



Data Elements : (i) Percent of 26 closures with weekly earnings at 

or above federal mi'nimum wage ^ 

(ii) Comparison- of earnings of competitively employed ♦ 
26 closures to earnings of other employees in 
state 

. VR's most basic purpose is to assist dis.^bled persons in finding gainful 
employment. One fundamental aspect of gainful employment is the ability to* 
be economically self-sufficient. > , 

■i • ' 

Data Element 4ri): Percent -of 26 closures with weekly, earnings at or above 
" federal minimum wag^ ^IjT ^ - " 

Rationale ' 

In. addressing Neconomic independence, the logical p^ate to look is to 
wages. This first data element as^^rSl/es wages as they, compare to the federal 
minimum wage. ' The normative implications of this data element are that a 
disabled person, should be expected, under equivalent circumstances, to make 
at'* least the minimum wag^ required by law. This data element uses the weekly 
minimum wdge figure as the standard rather than the hourly wage,. because the 
former more accurately captures the concept of this standard. IVhereas hourly 
wage indicates a measure ^*the employee's worth to tbe employer, total eat- 
ings is a better ind'itator of the employee's fii^anCial. well-^eing. If an 
employee is able to work &nly fivj^ours 'a week, his/her economic conation 
will be affected by this, as well as by the hourly rate. 



Formula 



Number of , 26 closures with weekly earnings ^at or above federal 

minimum wage^ « ' ' 

Number pf ,26 closures 

y 
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Data Sources 

• RSA-300 

• U.S. Census Bureau, Statistical Abstract of the U.S.* 



Jata Element ^-fii) : Comparison of earnings of competitively empLoyed 26 
^ closures to the earnings of other employees ifT state 

^Rationale 



This data element controls for state-to-state variation in earnings 
levels, whereas data element 4(i) does not, In some respects, this is a 
more comprehensive indicator than data element 4(i) because it provides an 
estimate of a client's standard of living relative to other persons in his 
or her state. - ■ , ' 

, Formula ' 

Mean weekly earnings of competitively employed 26 's 
« Mean weekly earnings of pther empld^es in state 

Data Sources 

• RSA-300 

• U.S. Census Pureau, Statistical Abstract of the U,S . 

T 

STANDARD 5: THERE SHALL BE MAXIMUM PLACEMENT OF REHABILITATED CLIENTS INTO 
* COMPETITIVE EMPLOYMENT. NON-COMPETITIVE CLOSURES SHALL REPRE - 
SENT AN IMPROVEMENT IN GAINFUL ACTIVITY FORtTHE CLI ENT 

bata Elements: (i) Percent 26 closures competitively, employed 

(ii) Percent competitively employed 26 closures with 

hourly earnings at or above the federal minimum wage 

(iii) Percent non-competitively employed 26 closures 
showing improvement in/function and li^e status 

Like Standard 4, this standard is concerned with the impact of VR ser- 
vices on their clients. As previously discussed, competitive employment has 
been seen as the best kind of closure. Recognizing that competitiA/e employ- 
ment may not be the appropriate placement for all clients, VR regulations 
require that any placement of a successfully closed client , whether in com- 
petititve, sheltered, or non-competitive employment, be rnto- ^'gainful and 
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suitable employment consistent with his/her capacities." For this reason, . 
improvement in gainful act'ivity for non-competitive closures is also 
included as a data element for this standard. 

Data Element 5(1): Percent 26 closures competitively employed 
Rationale 

. , — ^ / 

This standard's bias toward competitive employment reflects" the belief 
that vocational rehabilitation should focus on employment , .preferably compet- 
itive employment. For a standard emphasizing maximum placement into competi- 
tive employment, perhaps the most obvious data element is to determine the 
proportion of 26 closures placed into competitive employment. , ^ 

Formula ^ ^ 



Number of competitively 'employed 26 ^s 
Number of all 26 closii^;^s^ 



DcVta S(5urces 



RSA-300 



' Data Element Sfii): Percent of competitively employed 26 closures with 

hourly earnings at or above the federal minimum wage - 

Rationale • 

This data element applies more stringent criteria to the measurement of 
'•maximum placement of rehabilitated clients into* competitive employment.'' It 
compares the number of competitively employed 26 closures wit}\ hourly earnings 
at or above the federal minimum wage to the total number of competitively 
employed 26' clasures. As in data element 4(i), this data element implies that 
a disabled person in the competitive labor market should be expected to earn ' 
at least the federal minimum wage. Unlike 4(i^, however, this measui^e repre- 
sents an employee's worth to the employer, as determined by the client's hourly 
'Nfft^e. ^Thus, this data element provides a measure of the value of rehabilitated 
Vlf clients who- are in the competiti^re labor marketS-elrative to the federal mini- 
mum wage. 
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Formula 



Xumber of competitively employed 26 closures with hojurly 

earnings at or above federal minimum wage 

Number of competitively employed 26 's 



Data SourcQS 

• RSA-300 

• U.S. Census Bureau, Statistical Abstract of the U.S. 



Data Element 5(iii): Percent, of non-competitively employed 26 closures 

showing improvement in function and life status 

Rationale s 

Closures into non-competitive employment may be legitimate for certain 
clients, but in order to attribute any credit to VR for "rehabilitating'' 
Clients into non-competitive employment, there mUst be some indication that 
VR helped improve those clients' capacities for gainful activity. This data 
element will use information gathered on clients at acceptance and at clo- 
sure, usin^....ale^ents of the Functional Assessment Inventory '(FAI) and Life 
Status Indica^fors (LSI) instruments which will be added to the client's 
RSA-500. RSA is currently undertaking a- pretest of the FAI and LSI items ^ 
to determine which specific item^ to include on the RSA-300. 



Formula 

Xumber of non-competitive 26 's with improvement on LSI-FAI 

measures from plan to closure 

Number of non-competitive 26 's 



Data Source 



• RSA-300 
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STANDARD 6: REHABILITATED CLIENTS "SHALL. EVIDENCE VOCATIONAL GAINS 



Data Element-s: 



(i) Change in average e^^mings for^^^ closures 
(ii) Other changes in functional ability and life status 



It is axiomatic that rehabilitated clients should evidence spjne sort 
of vocational gains either in monetary or non-monetary terms at ^e point 
'VR services^ terminate. ' This standard assures that attention w4il be paid 
by the VR field .to the documenting and speking changes in a client's earn- 
ing status^ functional ability, or life status. It supplements the concern 
for rneas.uring post-service outcomes (as in Standards 3-S) by^jfsing the 
client's pre-service circumstances as a baseline for comparison.' 

') - ■ ^ ' 

Data Element 6(i): Change in average^ earnings for 26 closures 
Rationale 

This data element is included because wages are the most straight- 
forward indicator of vocational change. Weekly earnings are used to measure 
the change in a client's, wages which occurred during the period of time hp / 
or she* received VR services. , . 

Formula 

(The sum of closure earnings for all -26 closures) minus 
* (the sum of referral earnings for all 26 closures) * 
Number of 26 closures 

' Data Sourc-e • - - . ^ 

> • RSA-300 = - . , 

Data Element Other :>:changes in functional ability and life status 

Rationale 

.•In addition to vocational change -(as* measure^d by data element 6(i)),* 
the VR program also often acts as a change-agent in terms of non-vocational 
aspects of a client's life. As with the data elements associat.ed with non- 
.competitive employment closures (as in data element 5(iii)), the methodology 
for assessing non-vocational change requires further development before a 
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specific computation formula can be developed. Following RSA's pretest of 
the FAI/LSI data items and the selection of those items which will be added 
to the R-300, further refinement of this data element will be undertaken. 



STANDARD* 7 : REHABILITATED CLIENTS- SHALL RETAIN THE BENEFITS OF VR SERVICES 

4 

^ . s 

Data Elements: (i) Percent of 26 closures retaining earnings at' follow-up 

^ (ii) Comparison of 26 closures with public assistance as 

the primary source of support at closure and at follow-u; 

(iii) Percent of non-competitively employed 26 closures retain 
, ing closure skills at follow-up 

Vocational rehabilitation. programs, like all service programs, ideally 
strive to have the gains realized by their clients through program participa- 
tion retained over time. Job losses shortly following successful closure can 
identify serious short-comings in a program's service strategy and may point 
to an incongruence between program goals and individual client goals. Are, 
clients being ^'rehabilitated" on O^y a temporary basis, or are the gains 
achieved during the service period retained over time? This question has a 
great degree of importance to the over VR mission and thus a standard in this 
area is highly appropriate. Aside from* employment measures of benefit reten- 
tion, additional attention is given to expaijding the data elements for this 
standard to include non -employment measures. 

Data Element 7(i): Percent of 26 closures retaining earnings at f^ollow-up 
Rationale 

As noted, retention of benefits gained thrpugh VR services is very impor-" 
tant both to the individual client and to the overall effectiveness of the 
program. This data element looks at retention of wages earned as one of the 
most .important benefits obtained frpm VR. 



Formu la 

Number of 26 's with earnings at closure who retained or increased 
earnings at follow-up 



Number of 26 closures with earnings at closure, surveyed at follow-up 
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Data Sources 
• ^ 4 

• RSA-300 • 
^ • _ Follow-up Survey (merge with RSA-300) 

Data Element 7(ii): Comparison of 26 closures with public assistance as the 

primary source of support at closure and at follow-up 

* Rationale 

This data element provides a needed dimension in assessing benefits- 
retention for non-competitively as well as competitively placed successful 
closures. Here benefits are proxied by measuring the extent of the clients' 
use of public resources. By focusing on the degree to which there is a 
reduced need for public assistance, an emphasis is given to the economic 
self-sufficiency of the client in terms of stability or impr((vement . 

This data element requires a new definition ^of/*primary source of support 
where ^'source of support'' is broken into only two categories Cpublic versus 
private) and where primary *is taken to mean the source supplying 51% or more 
of a person's total monthly support. * 

Formula 

Percent of 26 closures with public assistance as the primary 

source of support at follow-^up ^ ' 

Percent of. 26 closures with public assistance as the primary 
. source of support at closure . 

Data Sources 
; RSA-300 

• Follow-up Survey (merge with RSA-300) 

Data niement 7(iii):- Percent of non-competitively employed 26 closures 

retaining closure skills at follow-up 

f 

Rationale 

Retention of functional and life status benefits is equally important 
as the retention of vocational benefits, particularly in the case of/fnon- 
competitively employed 26 closures for whom non-vocational improvemtn may 
be a primary benefit derived from^ participation in VR services. This data 
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element 'updates the information provided by data element 5(iii7, and will 
use the same FAI arid LSI data items used* for data element 5(iii), Howeverj-^n^ 
for the purposes of this data element, '.the FAI and LSI items will need to 
be modified ipto a form suit-able for self-administration .by the clients, 
via rhe Follow-up Survey. The specific items and their forms, will fe deter- 
mined after completion of the RSA's JAI/LSI pretest. .Once i^fiplemented,. the. 
data element will have the following formula and dat^' sources* 



Formula 

Number of non-competitive 26 closures retaining LSI/FAI closure 

skills ^ 

Number of non-competitive 26 closures surveyed af follow-up 



Data Sources t 

• RSA-300 , . 

Follow-up Survey (merge with RSA-300) 

STANDARD 8: CLIENTS SHALL BE SATISFIED V/ITH THE VOCATIONAL REHABILITATION 
PROGRAM, AND REHABILITATED -CLIENTS SHALL APPRAISE VOCATIONAL 
REHABILITATION SERVICES AS USEJEUL IN ACHIEVING AND MAINTAINING • 
THEIR VOCATIONAL OBJECTIVES . ' ] ~ 

Data Elements: (i) Percent of' closed clients satisfied with their' 

overall VR experience ^ -J 

(ii) Percent of closed clients satisfied^ith specific 
aspects of VR ^ ^ ' 

(iii) Percent of' ^6 closures juds^ng the services they 
received to have been useMil in obtaining their 
job/homemaker situation^ or in* current performance 

As an indicator of consumer appraisal of services, the standard on 
client satisfaction with vocational rehabilitation services has considerable, 
merit. Since client satisfaction f»olls usually offer a high degree of, support 
'for the program, this standard is viewed as having distinct political value 
in lobbying for expanded financial support at botlj the state and federal 
level. Complementing the political utility of a satisfaction measure is the 
inclusion of a client utility assessment in, the standard. The intent of ; 
this clause is to ensure that successfully closed clieaits assess the utility, 
of VR s&rvices, positively in terms of actually having ' contributecf to their 



82 



B.19 



getting a job and functioning in it. ^As a substantive rationale for the 

satisfaction standard, utility assessment offers a valuable entree for 

probing areas needing program improvement and for ensuring consumer involve- 

ment In improving the xesponsiveness of Vft services to client needs. 
^ • 

Data Element 8f i) : Percent of closed clients satisfied with their overall 

VR experience ' [ - , ' 

Rationale ^ * ' ' 

As one of the data elements of the origi^nal nine standards, retaining 
overall satisfaction as a measure of program perfoiinance has several advan- 
tages including: (if the procedure is in place; (2)\^developmental costs have 
already been absorbed; (3) it constitutes a composite measure of client satis 
faction which responds to legislative and consumer advocacy concerns; and (4) 
the dat.a show' some discrimination among closure statuses. 

Formula 

Number of closed clients surveyed who are satisfied with their 

overall VR experience [ 

Number of closed clients surveyed ^. # 

Data^ Source 

* ■ " ■ • ■ — *> 

• Closure Survey (merge with RSA-300) , ^ - . 

" \ - ' ' ' 'y ' 

Data '^Element' 8 fii) : Percent off closed clients satisfied with specific 

aspects of VR 

*' Rationale 

' This data element ettenlpts to gain a more detailed picture of client 
^satisfaction with specific key aspects of the overall VR process. In par- 
ticular, the aspects isblated for inquiry include, questions "about the client's 
counselor, the^physicaj/restoration services received, the job training ser- 
vices received, and/the jobj|lacement proces^s. Consistaqt negative asq|[ss.ment 
in any one o?^hese\nre^||pp^ be highly useful in guiding state evaluations 
"Und providing substantive input 'to programmatic improvements. 
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Formula 



a. Number of closed clients satysfied with their counselors ^ 
• Number of closed clients surveyed . - . 

b. Number of closed clients satisfied with physical restoration services 



Number closed clients surveyed 



Number of closed clients satisfied with -job training ^services 
Number of closed clients surveyed 

Number of closed clients satisfied with job placement services 
Number of closed clients surveyed 



Data Source 



• Closure Survey (merge with RSA-300) 



1^ 



Data Element Sfiiil: Percent of 26 closures judgitv^^^he services they ' 
\ received to have been useful in obtaining their 

' job/homemaker situation or in current performance 



Rationale 



Rehabilitated clienj:s can make fairly objective assessments of whether 
the services they received were instrumental in securing their outcome sit- 
uations. It is equally as important to assess thd^:ontribution VR seryices 
make both to the attainment of a specific ^closute situation or job and 'to 
the development of more general skills which help clients function in'' these 
new positions. \Vhile not unequivocably obiective, the client^s assessment 



of whether he or she uses the skills and/or knowledge gained from VR services 
is the closest approximation of the case. 



Formula 



N^umber of 26 closures jiidgi^ng the services they received to have ' 
been useful in pbtainilig their job/homemaker situation or in 

current performance ^ ^ 

Number of 26 closures- surveji^ed ~ \ * - 

Data Source ' ^ 

• Crostire Survey /merge with RSA*-300) 



s 
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COMPUTIXG TllE PERFORMANCE STANDARDS DATA ELB1ENTS ^ 

Having provided the reader with an overvuew of the Performance Standards^ 
and data elements, the next task is to provide the detailed information, needed 
to access the required data and compu.te the data elements. Table 2 serves 
this function. Reading from left to right, the table provides the following^ 
information for each data element: 0 

• the data element wording and the equation for computing the data 
element ; 

• definitions of terms used in the' data element's equation; 

• the sources (i.e.V, documents, reports, or surveys) which provide 
the information needed ^to compute the data* element ; 

• the data specifications, which identify the location of- the spec- 
•\ ific information items used 'to compute the data element.^ 

Table 2 should suffice as the general instructions on how to compute the 
data elements. However, there are two additional points which must be made 
regarding the process of accessing data and computing the data elements: 

1. iMergi^g of client 'surveys with client- RSA- 500 records : The client • 
.Closure and Follow-up purveys will need to be *'mergeB^" with the individual 
clients' RSA-500's. In the case; of the Follow-up Sun^ey, this required 
so that comparisons mav be made between the client's si'tuation^at closure 
(d.g., earnings level) and his or her situation at the ^pl'nt of follow-up. 
Tlie 'data items using the Closure Survey do not require afvy. over- time compar- 
isons. However, the Closure Survey should'be .merged with the RSA-SOO data • 
record thaF^RSA and state agencies -may have access to data on thie client's 
personal characteristics and services provided. In this way, RSA and state 
agencies may conduct policy-related analysis when problems in performance 
appear in the satisfaction/service utility data elements. For both the . , 



^The reader must boar in mind that these locations may change as a 
result of rev'j ion,s_ to RSA reports. 
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Closure and* the Follow-wp Survey, merging requires that a consistent identi- 
fier appear both on the RSA-300 and on the Survey. 'The client's case number 
or Social Security number are the most logical client identifiers to use. 

2. Using only 'Valid'' cases to co mpute data elements: Most of the dat 

T"^ ' ' ' ^ » 

el^^ffts for StatWards 4-8 require input of client-level data./^U calcula- 

tions must be made using only those cases for which "legitimqrte" data exiat 

(i.e., using only "valid'* cases). This excludes cases on whiph data are 

"missing," because: , ' ' 

• the^ counselor could not obtain the information for efitry on the 
RSA-300; 

• the client gave no response to a question on the survey; 

• the client could not remember or did not know the answer to a 
question on the survey; or 

• the question was not appropriate to the client's circumstances 
(e.g., clients receiying no physical restoration services should 
not be used 'to assess .satisfaction with physical restoration 
services)-' 

For most of* the data elements using client-level data, the valid cases 
will determine the denominator foB^TTie data element. For example, data 
elemenf 4(i) computes the percent \f 26 closures earning the weekly minimum 
wage at closure. Assume that thereVre 1,000 26 , closures, total; but that 
200 of those cases are missing data on earnings at closure (leaving 800 with 
"valid" data). Assume further that, of the 800 with valid data, 400 earned 
'the weekly minimum wage at closure. Depending on the denominator used, the 
state agency's performance on data dement 4(i) will vary: 

using all 26 closures;' 

400 (^ earning weekly minimum wage) 
4- 1000. ' (all 26 closures) * ^ 

^ 40% • ' . 



• 



using only valid cases: 

400 \(n earning weekly minimum .wage) 
T" 800 closures with valid d'ata) 



Clearly, in this case (and in fact, in all cases where a percentage 
score is computed) a state agency's performance will appear ''bettei:** when 
only valid cases are used for the^computation. Further, since we do not 
know the true situation of clients for whom data are misrsing, we -may mis- 
takenly bias the score downward when including invalid cases. (For example, 
in the numerical example above, the 200 cases with missing data may in fact 
have been earning the weekly minimum wage. Had the data been available, the 
agency's score would have been 60%. We must, however, assume that they we:re ^ 
not earning the weekly minimum wage, if we wish to include them in the cal- 
culation.) In short, because we wish to provide as accurate a picture of 
pei^formance as possible, based on the available data ', we must compute the 
data elements using ^only those cases for which all data exist ;^ that is, the 
valid cases. 




Table 2 

SjLimmary of Data Elements, Definitions, and Data Specif icati'Ons 
for the VR Program Performance Standards 



SrA.Sf)\Rb 1. Vni ATIONAl. RLHABI MTATION SHALL SERVE .THE ftAXIMUM PROPORTION OF 



POPULAnUN, SHBJLCT TO THE LEVEL OP FKDHRAL PROGRAM RITjDING /\ND 



ENT^IALLY ELIGIBLE TARGET 
TIES A^K3NG CLXENXS, 



\ 



n*t3 ri^^monr and foUatton , 


Definitions ^ 


Data Sources 


V J 

i 

iJata Specifications ' 


— ' ^ ' ) . 

(i>^ Clit-r.ts served per 100,000 
popuj.u loii: I < ^ 

* served' in a^iven year 


*, served = all active and 
closed cases (statuses 10-30) 
in the year . • 


RSA - U3 
(Oct - Sept) 


11. A. 3. a 


s<ate populatAon (in 100,000s) 
» 

' k 


State'populatiton = current 
best proxy for "eligible 
population," Divitle state 
population by 100,000 and 
trunca'te at two decimal 
points. , 


U.S. Bureau of 
the Census, 
Current Population 


State. population estimate 
a:s <of July ^ ' 


Reports, Series 


P-25 


(n) Percent severely 'disabled 
ser,ved: ' 

# severely disabled served 
in a given year 


# severely disabled served = ' 
all active and closed 
severely disabled cases 
-(statuses 10-30) in the year. - 


.RSA - 113 
(Oct - Sept) 


II.A.3.b 




* served in a given year 


1 served = same as l(i).. 


RSA - 113 
(Oct - 5ept) 


n.A}.3,a 1 
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* STANDARD 2: TT^C ^VOCATIONAL REHABILITATION PROGRAM ^iAll USE RESOURCES IN A COST>EFFECTIVE W NNCR AND Si'lOW 
A PObUlVK RLnmN TO SOClllY 01 1.SVES171LM IN VOCAnONAL RmiA^lLITATlON OT' DISABI.LO Cf ILNiS . 



h.Jt.i r.hiricni ami hjuntion 


definitions ^ 


Datp Sources 


Data Specifications 


(i) rxp<.nviiturc per conpetitively 
cTpiovcJ l(> closure: 

lt't,i] "Ji^cncy 'orofiJi tiircs 
* CO'. pet J tt vol y employed 
26 cloburcs ' 

. ' «• 

»f *• 


^ loial agency expenditures « 
all monies (stJto and federal) 
spent under control of .state 
agency '(110, Trust I'und, S;5I,\ 

. Un, and any otJier funds under 
control of state agenc>t). 

» competitively eiilployed 26 
closures a * of non-BCP self- 
employed and wage and salaried 
workers in the competitive 
labor market . 


RSA - 2 
(Oct - Sept) 

RSA - 300 

. \ 

• 1 


in.C.8 (Sec. 110) • * 

♦ IV. 4 (Trust Fund ♦ SSI)^ 

4 V.C.7 (UC)^ 

Item 4. P. 2 (26 closures); 
<^bdes I and 3 on Item 4.1 
(competitively employed) 


^ 1 

(ii) Ixpciuiiture per 26 closure: 

fotol af.cncy expenditures 
J^^^ closures 

, /* 


■ 

jTptal agency expenditure^ 
same as 2(i) . 

t 

^ 26 closures = closures 
during fiscal year. *^ 


R$A - 2 
RSA - 113 


III.C.8 (Sec. 110) 

♦ IV. 4 (Trust Fund ♦ SSI)^ 

4 V.C.7 imf , 

Itep II.A.4.a 


fill) P.-n lo^op total VR benefits to 
t,ot.Tl VR c</sts (benefit - 
t rdtjojc 

BuK^titsfc - 
Costs 1^ 


Benpfits - paid (jjirning'^ , 
^ ^ + .howcfTi.'iking mafietary • 
va 1 un t i on 
+ unp.iul work monetary 
valuat ion 

♦ fringe benefits ^ 

♦ change in labor 
force participation 

; , - repe«iter costs. 
Oiscj)unted present value. 

.Costs = total program cos^ts ' 
+ costs ^borrw by noni-VR 
parties ; * 
* '+ client-borne costs - 
'+ foregonq cl?ont, bene- 
fits (wages:and \\ 
fringe ber^efits) ' 
research costs 
4 training costs 
4 demonstration costs 
- t carryover costs a;ul 
■maintenance costs). 
Discounted present value. ' 


RSA - 300 
RSA - 2 
PSA- 113 

I'.oUbw-up Survey 
% 


Sdmmary of data requirements 
appears in Appendix 1. 
Model currently undergoing 
revi'siori. Final version 
oF model will be incorpora- 
ted ir^ (he MIS. Docuraenta- 
tioln on data requirements 
and nuithematic^l formulation 
of revised model will be 
^available from RSA . 
after final revision. 

V « 

1 ■ / 


(iv) lat.ii'nct benefit, from VR 
scrvic<i<; (discounted net* 
pr(;scnt valu(^) : ; i - * * ^ 

X 

. Benefits - Cests 


Benefits = sane t^s 2(lii). 
Costs»3 snmo as 2(iii). 

' >. — p- , — > 1 '■ -' — ■* 


RSA - 30Q 
RSA - 2 ^ 
RSA - 113 .{ 
I"'oUow-up SurViCy 


, Same a^ 2(iii) 

» 

1 * » 



^If the fund IS, r^levflf^t Jto current operations 
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SlANPAkD 3; VR SHAM. >L\XIMIZE TMb UU^^BlK AND PROPORI ION Or CLIENTS ACCKPTCD FOR SERVICES MtO ARE 

StiCC LSSHJLI.Y R UtABILITAKED, SMBJLLT.IO ilii. Mli.ilNG OF OnihR Sh\NUARI)S. 

-jvy— ^ 



er|c 



Pat.i 1 Icmcnt and Hquation 


Oef iiutions 


Data Sources 


Data Specifications 


(i) PerctMit 26 closures; 
» 26 closures 


* 26 closures = 26 closured 
during fiscal year. 


KSA - 113 


II.A.4.a 


M 20% 28' h 30 closures. 


* 26 +28+30 closures = 
total acceptedvClients closed 
(26 + 28 + 30) during fiscal^ 
year. 


KSA - 113 


n.A.4^a 
+ II.A.S.a 

+ II.A,6.a " ' . 


(ii) AnniK^l change' in number of 
2() closures: / * 

^ (#*. 26 closures in cur/ent year) - 
(» 26 closured ih prwious year) 


* 26 closures, current fiscal 
year. 

It 26 closures, previous fiscal 
year. 


RSA - 113 
(Cur'rcnt year) 

RSA - 113 
(previous year) ' 


II. A. 4. a 
II.A-.4.a 
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STANDARD 4: RmtNBILITATCD CLinNTS SHALL fcVIDENCE: ECONOMIC INDEPENDENCE. 



[<a'ta Lier.ent^nnJ r<|u.ition 



(0 Percent 20 cJosures uith weekly 
earnings at/above the federal 
minimiL'n wage : 

^ * 26 closures vith weel^ly 
earnings at/above the federal 
minimiim wa);e 
* 26 closures 



(ii) Comparison of earnings of com- 
petitively ejnpJoyed 26 closures 
to earnings of employees ^in the 
state: 

McMn wccKly earnings of com- 
pct itive I y emplo yed 26 's 
Meiin weekly earnings of em- 
p^loyees in the state 



Definitions 



Weekly earnings 
at closure. 



earnings 



Weekly minimum wage = 35 
hours X hourly minimum wage 
(BLS definition of full-time 
employment). 



* 26 closures = closures 
during fiscal year. 



Competitively employed 26s = 
wage and salaried workers 
(competitive labor market), 
♦and self-employed (non-BEP) , 



Mean weekly earnings. =r average 
earnings, week of closure, for 
competitive 26 closures. 



Hinployees in state = produc- 
tion workers in manufacturing^ 
industries . ^ 



Data Sources 



RSA - 300 



IJ.5. Bureau of the 
Census, Statisti 
al Abstract of the 



U.S. 



RSA --300 



RSA - 300 



RSA - 300 



S. Bureau of the 
Census, Statisti 




Data Specifications 



Item 4. J 



Federal hourly minimum wage 



Item 4.P».2 



Item 4. P.? (26 closures); 
codes 1 an5 3 on Item 4.1 
(competitively employed) 



Item 4. J (averag<j) 



Labor Force, Employment, 
and Earnings; Production 
Workers, Manufacturing 
Industries 

Uours and Gross Earnings, 
by state, average weekly 
earnings 



CO 
I 

ISJ 
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ST.\Nl>ARD 5: TflCRE SIbML BH MAXINfUM PUCDOT OF 'Rt l lABlIJTAYHr) CLIFtNTS INT(^ COMPETITIVE EM PtmMnNT 

KON-COfrlPbTlTlVE CLOSmUiS SHALL HLPRKSLNT AN lMi^Ktia:MENT IN (iAINI UL ACTIVITY FOOiiTCL'! FNT . 



Data Lletncrvt .iHd Lquntion 



(i) Fervent 26 closures competitively 
employed: 

1^ * competitively employed 26s 
* 26 closures 



1^ 



Perccirt compct 1 1 i v^fy em])Ioyed 
26 clo^iures with' hourly, earnings 
at/above the federal minimum wage 

^ conjpetitively employed 26 
closures with hourly earnings 
at/«'d)Ove federal minimum wage 
competitively employed 26s 



(ill) Percent non-com[/ctitivtly employed 
26 closures slio^S^ig improvement in 
function and life><^tus (imp^- 
ment after LSI/FAI 

* non-competi tive 26s With 
^ improvement on LSI/I-AI measures 

from pJnn to closure 

* non-coinj>etiti ve 26s 



Definitions 



^ competi^t I vely employed = wage 
and sal.iried workers (competi- 
tive labor market) plus self- 
employed (uon^BEP). 



•If 26 closures 



Hourly earnings = (weekly 
earnings at closure) 4- hcAirs 
worKed). 



Hourly minimum wage' 



^ competitively employed 2^s 
same as' r> (i ) . 



Data Sources 



RSA-300 



RSA-300 



RSA-300 



Statistical Ab- 
.sViict of the^U.S 

RSA.300 



Non-competitive 265 = 
sheltered workshop worker, 
self-emplo^'ed (BCP) , home- 
makers, and unpaid family* 
workers . 



Improvement on LSI/FAI = 
positive change cm functional 
and status iifdicators; measiires 
to be determined by pretesfR 



RSA-300 



R-300: 

• acceptance 

• closure . 



Data Specifications 



Codes I and 3 on Item 4*1 



Item 4. P. 2 



Item 4. J (weekly earnings at 
closure) J It^em 4.M {# hours 
Worked at closure) 



Federal hourly minimum wage 



Codes I and 3 on^tem*4.Ii 
Item 4.1^^2 



Item 4. P. 2 (26 closures)*; 
Codes 2, 3, 5, and 6 on 
Item 4.1 {non-corappti|t ively 
employed)" 



I 

to 

00 



Item 2.V 
Item 4.N 
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STANDARD 6:* RrULXDILITATlJD CLIKNTS SHALL UVIpHKCE VOCATIONAL CAINS . 



. 7 



Data Clemcfit uhJ Pquntion 


Definitions 


Data Sources 


Data Specifications 


(i) Avcrnge earnings change of 26 
closures, before versus after 
VR services: 

(Sura of closure earnings for 26 
closures) ^inus (Sum of referral 
enrninps foi 26 closures) 


Sum of earnings for 26 
closures = totnl earnings for 
the ^roup of 26 closures: 

• at closure <• 

• at referral. 


RSA-300k ' 
RSA-300 


Item 4. J' (sum for all 26s) 
Item 2.P''(sum for all 26s) 


* 26 closures 

> 


*26 closures = closures 
during ctirrent fiscal year. > 


RSAi300 


Iteih 4. P. 2 


(ii) pther chnnges in functional 
ability and life status 
(implement after LSI /PA I 
* pretest) 


Change' in functional ability 
and life status = same as , 
5(iii); measures to he 
determir^d by pretest. 


RSA-300: • 

• acceptance ^ , 

• cloM-^e 


Itbm 2.V' 
Item '4.N 




Table 2 (continued) 



STANDARD 7: 



UEHABILlTATiiQ CLIENTS SIIAI.I. Rf.TAlN THH BnNEl'lTS OF VR SERVICES . 
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Uata Hcmcnt and C^jnation 



(i) Percent 2o closures re'taining 
earnings at follow-up: 

' of 26 closures with earnings 
at closure who retained or 
increased Qarnings at follow-up 
V*i26 closures with earnings at* 
i^/osure, surveyed at follow-up 



(ii) Comparison of 26 closures with 
public assistance as primary 
source of support at closure 
and at follow-up: 

f * 26 closures with public 

assistance as pr-imary source 

of support at fol low - ij£ 

% 26 closures with public 
assistance as primary source \ 
of support at closure 



(ill) Percent non-competitively 

employed 2(i closures retain- 
ing closure skills at follow- 
up (implement after 1.SI/FAI 
pretest): 

* non-competitive 26 closures 
retaining I.SI/FA! closure skills 

* npn-competitive 26s surveyed 



at follow-up 



Def 1 ni t J<)ns 



Retained or increased . 
ea/^nings = cases where follow- 
up earnings arc greater than 
or equal to closuie earnings. 



I 26 closures with earnings 
at closure = cases where 
weekly closure earnings are 
greater than zero. - 



Only ^6 closures'are surveyed 
at follow-up. 



Public assistance as primary 
source of^ support = ca,se3 
where SSi , SSOl , AHH:, (;a, 
Workmen *,s Compt-nsation, and 
public institutions account 
(singly or in combination) for 
more than SOV of n*pe^son's 
total monthly support; 

• ft follow-up 

• at closure. 



Data Sources 



RSA-300 

Follow-up Survey 
(Merge with R-300) 



RSA-300 



26 closures (Only 20 closures 
are surveyed at follow-up). 



Non-cornpeti;tive closures = 
sheltered workshop worker, 
self-employed (BI:P), home- 
makers, and unpaid family 
workers. - 



Retaining closure skills = 
equal' or greater score on 
functional and status 
indicators nt follow-up, 
compared to closure: 

• 'closure (measures to be 
determine^ by pretest) 

• follow-up (measures to be 
determined by'prctest)» 



Fol low-up Survey 
(Merge with R-300) 

RSA-300 



RSA-300 



RSA-300 



RSA-300 (closure 
section) 

Pol low-up Survey 
(Merge with R-300) 



Data Sped fi cations 



ltei?7 .»! (Weekly closure 
ean\ings 

Question 3 (Wcelcly earnings 
at fol low-up) 



Item 4. P. 2 (26 closures) 

Item A.J (Weekly closure 
earnings) 



Questions 3, 4 : 
Q.2 4- (Q.2 ^ (Q.3x4) 

Item 4.N.1 



Q.4) 



Item 4. P. 2 



Item 4. P. 2 (26 closures); 
Codes 2, 4, S, and 6 on 
Item 4.1 (non-competitively 
employed) 



Item 4.N 
ft 

Question 



C3 

I 

o 
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Ucf ini t ion$ 


Data Ju)urces 


Data Sjieci f i^^at Jons 


(i) IVrcwit closed clients s;itisfied 
witb overtill VR ex|^crience: 

< c'loscd clients surveyed satis- 
fied with overall VR experience 
' cIos<?d clients surveyed 


Closed clients = 26, 28, and 
30 closures (Clostirc ^urvey 
given only to 26, 28, and 30 
closures). 


ClostJr^ Survey » 
(Merge with H-300) 


« t 

Questional (all respondents) 


(ii) Percent closed clients satisfied 
with specific aspects of VR: 

» closed clients satisfied with 
sj>cciflc aspects of VR 
* closed clients surveyed 
(four equations) ^ 


Closed clients = 26, 28, and 
30 closures. 

Specific aspects = satisfied/ 
not satisfied with counselor, 
physical restorarioii services, 
job training services, job 
placement services. 


Closure Survey 
(Merge with R-300) 


Questions 2 - 8 . 

CAI 1 respondents) * 


(ill) Percent 26 closures judging 

services received t* have been 
useful in obtaining their job/ 
hoiJiemaKer situation or in cur-^ 
rent performance; 

^ 26 closures judging services 
received to have been useful 
in obtaining^thei r job/home- 
m.'iker situation or in current 
performance 

* 26 closures surveyed 


Closure Status » 26. 

Useful « "useful in helping 
get or perforin in*' the 
person's closure occupation. 


RSA.300 

Closu>c-^rvey 
» (Merge with 11-300) 

I 

• 


Item 4. P. 2 

Question 9 (26 closures i 
only) 

• * 
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ADMINISTERING THE CLOSURE mX) FOLLOW-UP SUR VEYS 

The Vocal:ional Rehabilitation Performance St^d?rrds . require that *VR ' 
clients be satisfied with the services and training received, that the 
Services are useful in obtaining and peo^rming jobs and that the benefits 
of the VR program be retained. In the past, VR agencies have gathered in- 
formation addressing issues such as^these through the use*of follow-up 
questionnnaire surveys. However, an earl/ BPA review of the VR Program 
Evaluation Standards noted many criticisms of this effbrt including non- 
comparable sa^npling designs, survey formats,- definitions, 'and resulting 
data J across states; hi^h nonresponse rates.; and reporting biases. The 
closure and follow-up' surveys described here are designed to replace the 
current unstandardized system with a new approach, standardized across 
states. and yielding valid, useful data. 

^ I ; 

CLOSURE SURVEY 

; , V . . , . • . > 

^ The closure survey is designed to be distributed at case closure to 
clients whose service^f^are terminating. As a self-completion mailback 
questionnaire, it is designed to be self-explanatory. 

The closure 'survey wilT sejve.as the source 'of data about client 
satisfaction with overall services and various aspects of services ^<e.g. , 
counselor performance 'the quality of placement services). It ^Iso will 
provide data on the clients' assessment of the lisefulne&s of their services 
V in obtaining and functioning in their job or homemaking sit^uation (see 
Figured). ^ ' * ^ - . - t ' 

Both the clos\ire and follow-up surveys are essential sources of data 
,for computing specific data elements. Table, 1 shows' how' the closure 
surv^ relates to the VR Performance St^andarji^ V indicating which items 
on the survey are tised to provide data for specific data elements. As 
with the other w standards and data elements, these are useful not only for 
measuring, total agency performance, but also as a closer look* at district , 
and evQn individual counselor performance. . The s^losure syrvey can be used 

erIc • " ' loi 
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Figure 1 
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Closure 
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Rehabi 1 i tat ion Process 
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FOLLOW-UP SURVEY 



Benefit's Retained? 
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• . * Table' 1 

Uses of Closure Survey Data t-or< Performance Standards 



Closure Survey Question: 


Used m Data. Element: 


1 


8(i) (Percent of closed .clients satis- 
fied with overall VR' experience) 


2 


— 7i — * — - 

oi±xj (^rercent or closed clients* satis^ 
fied with their (Counselors') 


3. 4 , 

•* 


8(ii) (Pe?:cent of closed clients.re- 
^ ceiving physical restoration 
services, who are satisfied with 
those services) 


5, 6 


8Cii) , (Percent of closed clients re- 
ceiving job training 'services , 
who are satisfied with those . 
services) • ' ' 


- 7, 8 ' 

» 


■ 'V • — ^ — 

8(ii) (Percent of closed cl,ient§ re- 
ceiving job placement services, 
who are satisfied with those 
services) ' ' . 


9 

V 


'8(iii) (Percent of 26 closures judging 
services receivecl to have been 
useful ^in obtaining their job/ 
homemaker situation or in current 
performance) 

' — n ^ " — 1 
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to help identify strengths and wea"knesses in specific service areas. Also, 
by linking this data to data on individual client characteristics, it is 
possible to study satisfaction with services'and utility of services across « 



different client groups. 



FOLLOW-UP SURVEY 



I 



The follow-ui5^ survey is designed to be distributed as a mail -back 
questionnaire to be completed by former clients one' year following com- 
pletion of VR services. The major purpose of the follow-up survey' is to 
determine the extent to which the benefits gained by the VR /client have 
a long-term effect. This is accomplished by asking clients about work 
status, earnings, and otlier sources of support, and about a variety of 
skills, to determine to what extent the benefiT^ 'of services have been 
retained over a 12*-month period, - ,* 

Once again, a table has \een provided illustrating the relationship 
between items on the survey' and specific d^ta elements of the standdrdS^. 
Also,*as in the case of the closure survey, the follow-bip survey can- pro- 
vide valuable information at the district and^ counselor levels in addition 
to proxiding the necessar)' data for computation of the standards. Through 
ihe follow-up surveys, in conjunction with data about individual client 
characteristics, VR can learn much about the long-Uerm impact of the 
program's services on different types of clients, as well as how that 
impact might vary across districts or counselors. 
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Table 2 / . • ■■ 
Uses- of Fo;iow-Up Survey Data for Performance Standards 



■Follow-Up Survey Question: 


Used in' Data" Element : ' 


1 . 


(Update -on work status. Not used 
explicitly for standards.) 


2 


7(ii) (Primary source of support) 


3 


2(iii), 

2(iv) (Benefit-cost) 

7(i) ' (Earning^ retention) 

7(ii) (Primary source of support) 


4 


7(ii) (Primary source of support) 


5^ 


— — ' 
7(iii) (Retention of 'functional 

abilities and life status) ""^^ 



Questions to be added after pretest of the Life Functioning Index (LFI) , 
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ADMINISTRATION OF THE .SURVEYS : PLANNING -AHEAD 

r \ ^ - 

Having presented a basic overview of what the closure and ^follow'-up 
r surveys are, the' next sections of this guide will describe methods of 
effectively implementing the surveys. The first step in this* process is 
' to plan ahead* This may seem at first like stating the o'bvious, but the 
importance of planning for the administration of the surveys cannot be 
adequaxely stressed. Just as error can be re<iuced .through appropriate 
samjjling methods or questionnaire design, so it is also possible to reduce 
error through adequate planning. For example, one would be unable to 
implement a carefully timed follow-up procedure requiring replacement 
questionnaires if too few are printed. Likewise, processing returned 
questionnaires and follow-up reminde:ps requires planning in advance for 
adequate staff. Attention to administrative details may be as crucial 
to obtaining high quality data as the questionnaire or cover Jetter. 

There are four major steps in planning the implementation of the 
surveys : 

* 1) identifying all tasks to be accomplished; 

2) determining how each task is dependent on the others; 
. 3) determinii)g in what \prder the. tasks must be performed; and 
4) deciding the means by which each task is to be accomplished, < 
IWiile it will not be possible here to take the reader through all of 
these steps, the sections that follow will identi^fy the major tasks and 
provide methods for accomplishing them. 

There are five major activities involved in implementiag the siprveys 
and collecting the data: 

• Sample Selection; 

^# Instrument Development; 

• Survey Distribution; ' . , 

• Survey* Collection; and , ^ 

• Cleaning^ and Coding the Data.- 
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Each one of tKese activities, or tasks is essential to collecting qual-ity 
data. . Perhaps these tasks can best be viewed as links in a chain. If 
any one of the links is weak, the whole chain' is weakened. Jn, order to 
ensure an effectively administ^re^ survey, each one of ^he^e activities 
must be. planned ahead. The following pages describe the components of 
these activities and the recommended methods. for carrying them out. 
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SAMPLE SELECTION 



. Tlia careful selection of an appropriate survey sample is crucial to- 
the design of any survey or other da^a collection proces.s where data is to 
be collected from l.ess than the total universe of potential respondents. ' 
Obviously, the best Way to -prevent sampling biases or other sampling prob- 
lems wQuld be to distribute closure and follow-up surveys to' alUclients. 
However, not only. would all* but the smallest states find this impractical, 
but it is^also not necessary. * For the purposes of this survey, a randomly 
selected sample of appropriate sizesvill provide adequalje .data to respond 
to the VR Performance Standards as well as provide state -VR agencies with ^ 
a valid base for collecting information about any other aspects of the pro- 
gram that the agency 'may choose to include in the* surveys. ; 

Who Should Be Included in the Sample? " 

The closure survey is designed to collect data from clients as they 
are closed from VR' services. Since thfe closure survey is designed to 
gather information about clients' satisfaction with services and the 
usefulness of those services, clients closed in Status 08, having received 
no services, will not be surveyed. Thus, the sample will include: 

• 26 closures, , . ' 

• 28 closures, and 

• 30 closures. 

* 

The follow-ui) survey was designed with a different purpose in mind. 
Rather than surveying a:ll clients who received services ' from VR to find 
out about their experiences, the follow-up survey is designed to determine 
whether the benefits received have been' retained ov^r time. For this sur- 
vey, only those receiving substantial benefits from their VR services are 
included. Thus., the sample will include only 26 closures. 
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How Many Clients , are Needed for the Sample? 

For the closure ' survey-, the sample should include 500 clients each 
year. This should be distributed amoi% the three closure statuses as 
follows: • ' ^ — 

• 26 clostires - 300,/" 

• ^8 closures - 100, ^ 

9 

• 30 closures - 100* ' 



Clients should be sampled on a monthl^basis throughout the year. Thus, 
the sample size for each closure status will be as follows: ^ ^ 

• 26*^closures - 25 per taonth, » [ 
r 28 ^jl^ures - 8-9 per montJi, . 

• 30 closures -^-9 per month. 

For the follow-up survey', the s^ample should include^ 200 clients per 
year. .As in the ^case of the closure survey, clients shc^uld be sampled 
throughout the year pn a monthly basis. Thus, the sample" for the foUow- 
up survey will include 16-17 26 closures per month'. . . 



The'Monthly^Sampling Plan ^ 



In order td ensure the best possible survey data,, it is important that 

the selection of clients be made on a random basis. Each » month the sample 

should be draWn by systematically selecting every '*n" case. The ''n" here 

is determined by dividing the total number of cases by the number to be 

selected.^ For example: * 

Actual # Monthly .Select 

Status of Cases Sample Size Every 

26 250 

28 99 

30 78 



25 • 10th 
" 9 11th 



9 8th 

A new monthly sampling plan must be made each month. To facilitate 
this process, sampling forms for an entire year follow. 



/ 





t 

Mortth ' 


Monthly Sampling Plan: 

7 

Month 


Closure Survey 
« 




Statu*; 


Actii.il 
of C.isos 


•Jail t hi) 
Si:o 


Sc 1 oc t 


. Actual 
iVumber 
0/ Cases 


Monthly 
Saraj) I e^ 
Si ze 


Hvery ^ 


•Ntm t n 

Actual 
Numhof 
of C.iscs 


Monthly 
SamiUe 


Select ' ' 
Every 


Month 

Actual 
Nurcber 


Monthly 
SaRip 1 (? 


Select 


26 
28 
• 30 


% ♦ 


: 9 


Ch 
th 




: 25 

T 9 

T 10 


. th 




z e 
i 25 

T 9 

i 9 


• th 


of Cases • 

' c 


Si ze 
•f 25 
* 

f.' 9 
: 9 


jA'Cry 

th 


th 


th 


th 


/ 

th 




th 


th 


t 













Status 


Month 

• Actual 
Number \ 
of Cases 


Monthly 

Sample 

Size 


% 

^iect 

Lvery 


Month 

Actual 
Number • 
of Cases 


Mi/ithly 

Sample 

Size 


Select 
Overy 


1 

Month 

Actual 
Number 
of Cases 


1 Monthly 
Sample 
Size • 


9 

V 

i 

Select 


Month. 

Actual 
Number 


Monthly 
Sample 


/ 

Select ' ^ 


26 
28 
30 

1 




' J 2S ' 
i 10 
^ 9 


th 
th* 


♦ \ 


r 25 

r 9 
r 9 . 


th. 


■4 


T 25 

r 10 
r 9 


Hvery 

th 


of Cases 


•Si ze 
. r 25 
t 

T 9 


f.very 

th 


th " 


th 


th 


th 


th 






1 


Month ; 


* 

Month * 


0 






P 




Status 


Ac t ua 1 
Number 
of Cases 


Month ly 
Sample^ 


. Jf-r- 

Select 
hvery 


Actual 
Number, 
of Cases 


Monthly 

Sample 

Size 


Select 
Hvei^ 1 


i^iunin 

Aetna 1 
Numbe r 
of Cases 


Monthly 

Sample 

Size 




Select 
Every 


Month 

Attudi 
Number 
of Cases 


Monthly 
Sample 


Select 


26 
28 
30 




i 25 
i 9 

T 9 


th 




T 25 
T 9 • 

r 10 


th 


t V . 

• <3 


7 25 
* 

^ 9 


th 


/"* 


Si zi: 
T 25 

r ^ 9 
f 9 


Every 

th' 


th 


r 

th 


^th 


' th 


th 


th 


th 


th 



n 



llg_ • ^ ■ ^\ . . " 113 



C-12 

A 

INSTRUMENT DEVELOPMENT 

The next major link in the survey chain to be described here is that 
of^preparing the questionnaires for distribution, fhe size of this task 
will vary greatly from state to' state, depending on- the extent to which 
agencies choose to modify Ihe standardized questionnSttes' to meet .the 
state's ofim needs. The preparation of- a questionnaire consists of tht^ 
major acti'vifi^s: 

^ 1)' Writing the questioi^; ' * ^ v, ' ^ 

2) ^developing the form*; and 

3) printing the questionnaire. 

Berkeley Planning Associates has already done the first' two of these steps. 
%k surveys appear at the end of this chapter qn- pages 3^5-33. Care- 
fully review the proposed surveys. If" there are no additional questions 
your stat^ VR agency wishes to ask, arid if there are no desired format 
changes, the first two steps may h& skipped and you may proceed directly 
to^tep 3 — printing the^ questionnaire. For those agencies wishing to 
modify the surveys, howevei*-. Steps 1 and 2 are described Selow. 




Step 1 - Writing tjie Questions " , ^ 

/The survey developed by Berkeley Planning Associates includes all of 
the questions necessary to collect client survey data for th« VR Perform- 
ance Standards-. IVhile agencies may choose to add ^ditidnal quetstions, the 
closure and follow-up surveys must contain at least those questions included 
in BPA's questionnaire. It is also important that all states ^se the same 
wording of these questions and ask them in the same order to standardize 
the responses sb that data can be aggregated and compared across states. 
Any questions that your agency may wish to' add must be added to the en4.of 
the surveys, fallowing the standardized questions. 

• Some states may c^os% to take advantage of this survey contact with 
clients to ask additional questions tj^at relate to the state's o\>m program 
evaluation and planning needs. \Vhen writing questions for a survey there 
are a number of consideratiams to keep in mind. 
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Identify the Kind .of Information Being Sought 

^ 9 

The first step ^n writing a question is to identify exactly what kind 
of information is desired from survey respondent's. Questions^ can usually 
be classified' as recfuesting one or more o:^ these types of information: 

• . attitudes what people' say they want; / 

• Beliefs — what people^ j:hink is true; 

• Jbehavior what people do; and 



• attributes — what people are. 

It is, important tS^ distinguish between these types of information and ^to 
determine which of these is the most appropriate for anslering ^iven P 
research, or evaluation question, 'otherwise, ef4rts to write questions 
may inadvertently result in obtaining a different .type of information f»om 
that which is desired. , ' ^ ' 

Decidin g Question Structure 

— ^ \ . / * 

The second major decision in writing questions is to determin/question 

strueture.f «}he basis for " distinguishing among question structures is the 

nature of the response behavior ask^d of the client. 

• Openr-ended — respondents create theif own answers; 

/ '• Closed-rended with ordered choices -- resjiojidents choose the 

most appropriate resg^se along a scale; •« , ■ " 



^ • ' Closed-ended with uiA)rdered choices 
^ frpm among discrete categories; and 



respondents .choose 



• partially closed-ended -- although responses are provided, 
respondents have ^he o ption of creating their ow n-. 



Eacji type of information mentioned earlier (i.e., attitudes, beliefs , ^behav- 
ior, and attributes) can be requested using any of these structures. However, 
some structures tend to be more suitable for obtaining certain types of 
information than others. Open-ended questions are useful ^or allowing res- 
ponde;its to express themselves freely or to elicit a precise piece of 
information where the number of possible responses is so larae that lasting 
them would be unwieldy. Closed-ended questions with ordere/ choices are ^ 
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suited to determining such* things as intensity of feeling, degree o^f involve 
menli^ and frequency of activity, Cl2;sed-ended-questions with unordered 
choifc^s are useful for asking fesponde]>£s to evaluate elch choice individ- 
ually. Partially closed-dnded questions provide an opportunity to give a 
response that' the' question-writer may. not have thought of. ' 

Choosing the Wording ^ ♦ • ^ / . ^ 

Having Secided what^to ask ^d how to ask it, the third decision is 
how to wofd it. .Effective wording of survey questions is a skili^that is , 
developed through experience/ However, there are a number of considerations 
that can help even a novice to state the question in a meaningful way. The** 
following qiiestions can serve as a useful guide to developing survey ques- 
tions : * . ' ' 

1. Will the woi?ds be uniformly understood? 

2. Does the question contain abbreviations or unconventional 
phrases? • ' \ j 

^^'^ 3. Is tlie question too va^ue? 

4. Is the. question too precise? 

5. Is -the question biased? 

6. Is the question objectionable? , 
^ 7. Is the question too demanding? 

8. Is it a double ques.tion? ^ 

9. Does the quest ionHfiave a double negative? 

10. Are the an^er choices* mutually exclusive? 

11. Have yon assumed t'oo much knowledge? 

12. Has too much been assup/ed about respondent behavior? 
J3. r Is the question technrcally accurate? 

14. *' Is an appropjfdate time referent provided? 

15. Can the responses be compai'ed with existing information? 
"16. Are the questions too cryptic? 

The use of this list as a checklist in writing survey questions can help 
the writer avoid Sie^ most common pitfalls in wording questionnaires- 
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I^tep 2 - Developing the Format ' / 

/ Once the questions h^ve been written, theVext 'step is f» figure out 
how to lay them ouT^on the pa^e^and tie them together in such a way that 
the questionnaire. can be self-explanatory and as easy as possible to com- 
plete. IVhile it is not possible here to dekeribe in detail layout c^^esign, 
there are a number of practical considerations that should be taken into 



account 



provide dirfe^ti^ns for how to answer questions; 
show clearly how to skip questions (if applicable); ' % 
establish a Vertical flow to reduce the likelihoad that 
questions will be missed; 

use different typeface for questions, answers, and instruc- 
tion^; ^ . . ' . 

pre-code the responses to the maximum ejctent possible; 
make questions fit comfortably on each page; 
use transitions .for continuity; and 
consider carefully the order of the quest ic^jj^^ 

, Step 5 " Printing the Que^ionnaire ' . 

The client's first e^osufe to the lopk and feel of the questionnaire 
may be critical to ensuring a good response rate/ It is very important 
that the questionnaire make a good first impression. It mustn't look too 
bulky, long, formidable, disorganized, or difficult tp complete. Take 
into account the following: 

• Make the questionnaire legible 

k&e white or off-white paper; - 

use printing method that produces quality very close to 
the original ; • ' i - 

r- use large type for visually-impaired; and 

i? using photo reduction, don't reduce smaller'than 
three-fourths of the original size, 

• Consider printing the questionnaire in booklet format 
— combined with quality paper^nd printing (this looks 

very professional); and^ 
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--^use 8-1/4" X 12-1/4" paper folded in half, as this 

V 

' • will fit in a conventional envelope and keep mailing 

costs down. 

• 'Don't print questions on the front or back pages ("cover" 

^ pages) ; andf^ ' . - ^ 

• Make sure to make an adequate number pf copies take 
into account follow-up mailings. 
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SURVEY DISTRIBUTION 



I eNiover 



Once the questionnaire has been prepared for distribution the/re are 
still a few additional tasks involved in aqtually getting th« survW in the 
mail.^ It is important that the survey be accompanied by a per^nablc 
letter from the VR agency" V» the client, explaining the purpose of the 
survey. It is. also important that appropriate ree/ord-keeping procedures 
be established to ensure accuracy in administering the survey. Finally, 
the surveys must be packaged and mailed out. ^ 
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Step 1 - Preparing the Cover Letter 

The" cover lettel* should be the first part of^ mail out package that 
the client will see. It serves to introduce th^e survey and hopefully 
motivates the respondent to immediately pick up the questionnaire, fill it 
out, and return it. _ The cover letter is virtually the only opportunity for 
anticipating and responding to'^respohdent questions. It must be short and 
to the point while still providing adequate information (see Figure 2) . 

The first paragraph of the cover letter should (1) explain what the . 
survey is for and (2) convince the client that the results will be useful^ 
The second paragraph should be used,j-to convince the client that his or her 
'response is important. Subsequent paragraphs should address confidentiality, 
tell the respondent what to do if questions arise, and extend appreciation 
for participation. , • - 

In addition to the body of the letter there are a few additional 
details that can help increase the re'sponse rate and are worth considering. 
The first is that dating the letter gives it an a-ir of greater importance 
and gives the respondent a frame of reference of elapsed time should it 
become separated from the envelope before being filled out. Secondly, it 
is best to type the name, address and salutation (if used) individually 
onto each letter so that it appears persiafialized rather than as a typical 
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Figure 2 
SAMPLE COVER LETTER 
(Closure Survey) 



Date : 
Dear 



In order to improve the effectiveness of client services, VR is 
asking for the opinions of some of our past VR clients. Please 
find enclosed a brief questionnaire regarding your feelings towards 
VR counselors, and the services you have received. The results 
from this survey will help guide the modification of VR services 
to make them more responsive to client needs. 

Since only a small number of former VR clients are receiving this 
survey, each person's timely response is of great importance. Use 
the enclosed preaddressed envelope to make returning the, completed 
survey easier. 

All responses to the , Purvey will be confidential. If you have any 

questions regarding the survey, filease carl ; 

at or write: 



VR is committ/e^to providing isefuKand beneficial services to all 
VR clients. Your help in this effort is greatly appreciated. 

Sincerely, 



. ■ . I2u 
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form letter • Finally, -the letter should be printed onto agency letter- 
head, (or a very good copy), ^ 



Step 2 - Setting Up the Records • , • ^ • : 

* • - " \ • . • . . 

Effective administration of the surveys ref^uires that the required 

information be recorded systematically in an 'easily accessible formi The 
attached 'Survey Control Sheet provides a simple Whanism for recording* all ' 
of the information needed during .survey distribution (and coll^ctionO in one 
place • • 

The first step in setting up the records is to compile a listing of" 
all members^ of the sample with up-to-date addresses and telephone numbers,- 
This information is recorded on the control sheet, one for each client, * 
Having this information on the control sheets makes it readily available - 
for follow-up response prods later and also provides a place to record 
changes in addresses and' phone numbers should they arise, . t 

The next step is to clearly identify each client by recording Social 

Security number and case number on the control sheet,- This is important 

for ensuring accuracy since, to protect confidentiality, no identifying 

information will appear on the questionnaire. 

As the questionnaires .are sent out, the question naire number frdm the 



survey sent to each client must be recorded on the control sheet to identify 

each questionnaire with a given client. This provides the identification neces 
sary to merge survey data ,With other agency data making it possiBle to look 
at the survey data in the context of client characteristics. This is also 
the only way to keep track of which clients have returned surveys; informa- 
tion that is essential for follow-up purposes. Also essential to the 
follow-up lA^ocedure is the recording of the date mailed out in order to 
keep track of elap'sed tijne. 

This is the last of the preliminary step^ Once T^is record-keeping 
system has been set up, the surveys can be packaged aijd mailed out 

Step 5 - Packaging and Mailing Out the Survey 

■ The survey package should include the Survey, a Qover letter, and' 
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Questionnaire No. 
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(Number on Survey Control ^heet and 

" Questionnaire must correspond) 

» 

^ • VOCATIONAL REHABIL ITATION CLOSURE SURVEV 

Survey Control Sheet • * 
To Be Completed i By Agency Personnel Only 

I. Agency 'and Client Identification 

-1. Case No. /<////// /.^ / / ' 2. Closure Date / /• 

3. Social Security No. / 7 / / - / I I - I I I I I 

4. Client's Name^ ; . ^ 



5. Client's Address 



street name and number , Apt. No. 

t 



city state, ^ zip code 

6. Client's Telephone No. ( . ) ^ , ' 

. ^ J ^ 

II. Survey Control Information (Check the bo^^es that apply and give the dates) 



I. Initial Questionnaire Distribu^d Date / 7 / 
How: by counselor? 



/_ 


J 


L 


J 


!_ 


'j 



Response Prods : 

2, Reminder postcard / / ' Date' / III 

3. Second survey form 
•4. Telephone follow-up 
$♦ ^Other: 



/. ; 


Date 


/ 


/ 


/ / 


/ / 


Date 


/ 


/ 


''/ / 


• /" / 


, Date 


/ 


/ 


±^ 












/ / 


Dat| 


/ 


/ 


1 1 


Date / 


/ / 


/ 







A 



Hdw: by mail? 



by telephone? Interviwer 's name:_ 

in 'person? 
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a return addressed envelope. The return envelope should preferrably be 
postage paid, as this tencjs to help maximize the response rate. Once 
again, the importance of r,ecording questionnaire numbers on the appropriate 
survey control sheets- cannot be overemphasized. This is usually the las.t. 
step in the mail out process in conjunction with the envelope stuffing 
process. The materials should be folded and the envelope stuffed such 
a manner that the respondent notices the cover letter first. 

Experience shows that to ensure that mail reaches its destination as 
soon as possible after the mail put 'date, the, best time to mail the surveys 
is early in the week. This avoids the weekend build up. It is also helpful 
to avoid holidays a^ much as possible. 
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SURVEY COLLECTION 



Most people who answer questionnaires do so almost immediately after 
they receive them, A questionnaire that lies unanswered for a coupl^^ 
weeks is not very likely to be returned. There are many reasons why 
respondents may not return the questionnaires: 

• The questionnaire ijeveV reached its destination because the 
wrong address was used. 

f • The questionnaire arrived at the appropriate address but was 

discarded without being opened because it resembled '*junk*' mail. 

it 

• The respondent found no coijvincing explanation about why it 
should be completed so threw it away. 

• The client decided ^o fill out the questionnaire but temporarily 
laid it aside and just never got back to it. 

• The questionnaire was filled out but the return addres's was 
misplaced so it was never returned. 

• The client was unable, because of his or her disability, to fill 
out the questionnaire without assistance. ' 

V/hiie some of these problems can be overcome by carefully preparing 

.the survey package, there will always be individuals .who do not return 

the questionnaire. For this reason it is important to follow the surveys 

with reminders or prods to those who have not responded. Berkeley Planning 

Associates recommen4s the following four step process for maximizing. 

response rate. Asj^ill be_seen^ ^ tju5_is^e^^^ 

• * 
# » 

will be an effective tool. . 



Step 1 ' Two Week Reminder 

Two weeks after the initial distribution, reminder post cards or^--^ 
letters should be mailed to clients for whom a questionnaire has not been 

12i 



returned (see Figure 3). The reminder should be written not to overcome re- 
sistince, but rather to jog memories and to influence the respondents' pri- 
orities. It should be carefully. worded to convey a* sense of importance without 
sounding impatient. This can be accomplished by stating that a, questionnaire 
was sent, stressing why the respondent is important, offering a replacement 
questionnaire, and providing a contact person. 

To keep costs down, reminders should be sent only to those who have 
not responded. IVhile some states may choose to use computerized flagging 
systems, this can be accomplished mechanically and quite simply through 
♦use of the control sheets. IVhen questionnaires are returned by clients the 
date is recorded on the control, sheet and*that control sheet is pulled from 
the pending file and filed by closure^ date. All those remaining in the 
current month pending file at the end of the two week period would receive 
a two week reminder. 

Step 2 - Second Questionnaire 

One week after the two week .reminders have been sent out, a second 
questionnaire should be maile4 to the remaining non-respondents. Onqe again 
this, should be accompanied by a cover letter (see Figure 4)* Tjiis letter 
should combine element^ of the first cover letter and the reminder by 
stating that a questionnaire was mailed previously, thanking those who » 
have already responded (in case this reminder has crossed in the mail 
with the response), explaining why the survey is important, stressing the 
importance of the individual's response and providing a contact person. 
If, as in the case of ^he reminder letter, the control sheets for those 
^who have returned questionnaires have been pulled and filed appropriate lyj 
then those ren^aining in the pending file at the end of th6 third week 
would receive the second questionnax-re . 



Step 5 - Telephone Reminder 

^ * 

One week 'following the second Purvey mailing, a telephone reminder 
should be undertaken for those surveys still unretumed. Once again ^ 
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Dear ^ 



Date: 



Our effort to refine and improve VR services has included 
sending brief questionnaires to selected former VR clients 
Such a questionnaire was recently sent to you and we want 
to emphasize how much we would appreciate your i^esponse. 
We encourage you to complete the questionnaire and return 
it to us in the preaddressed envelope as Soon as possible. 
If you have questions or need a new questionnaire, please 

contact \ at ;_ 

or write : 



Thank you for your help. 
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Figure 4 
SA^^PLE. REMINDER COVER LETTER 
(Closure Survey) 



Dat^ 
Dear 



VR is committed to iinproving the effectiveness of client s 



ervices 



and IS asking for tliS opinions of past VR clients. A questionnaire 
was recently sent to you regarding your 'feelings towards VR services 
and counselors. Since the results from the survey will help guide 
the modification of VR servic>s, your response to flie questionnaire 
IS of tremendous importance. 

Ple^ase fine? enclosed another copy of the questionnaire. We encourage 
you to complete the quesjionnaire and return it to us^ in the enclosed 
preaddressed envelope. All responses to the survey will be confiden- 
tial. ' 

•If you have alr^ady returned a completed questionnaire, please accept 
our gr-atitude. ^ If you have questions regarding the survey, please 
call ^ ' • ^ r . 

or write : ^ ^ 



at 



Thank you very much for your help* 
Sincerely, 



s 
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respondents should be urged gently. ^ The focus of the telephone corttact 
should be to determine the cause of the delay and establish whether the 
respondent is willing to participate. For those clients who have difficulty 
completing the survey themselves it may be necessary to make special 
arrangements. . ' ' 

Step 4 - Last Try ^ ^ 

If no response is received after Step 3, the agency may elect to 
mail a third questionnaire (if necessary) or to complete the survey through 
telephone or personal interview with the client. The latter should be 
arranged at a mutually convenient time and place for the parties involved. 
The "last try*' is the step most often omitted in survey implementation, 
especially where resources are limited and the i^sponse rate has been 
relatively high. In-person and telephone interviews take time and are 
more costly than self-administered mail-back surveys. However , .omission 
of this step increases the tendency for sample bias% Therefore, the 
extent to which an agency will implement "these procedures should receive 
careful consideration. ' . ' ' • 

Surveys still unreturne^ or uncompleted two weeks following -the 
telephone contact should be considered non-responses and classified as* 
s,uch. This all9ws for a total of six weeks for completing the .questionnaire 
before it is classified as a non-response. 

I " ^ 
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DATA PREPARATION 



• .The return of the questionnaires to the VR agency is not the 
completion of the survey/data- collection process* Rather, the final link 
in the chain is the preparation of "the data for analysis. -Before the data 
Can be analyzed, the questionnaires must be edited • 

Step 1 - Editing Completed Questionnaires 

» 

All completed questionnaires must be edited to ensure that the 
proper information is being collected, to ensure that the questionnaires 
have been properly completed, to ensure that information is recdtded con- 
sistently and to prepare the data fot keypunching. 

In the process of editing, the editor reviews each and every item 
in the questionnaire. He or she examines all responses to make sure that 
the instructions were followed, that the answers are appropriate and that 
the appropriate number of responses is given to each question. The agency 
should strive to have only one or two individuals edit completed instruments. 
By minimizing the number of editors the consistency of fhe data is increased. 
Also problems or trends in interpifeting or answering questions are quickly 
detected. ^ ^ 

Editors also keep track of items with a high degree of missing data, 
and items with which respondents have difficulty. Knowledge of problematic 
variables is very important in the analysis phase. 
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Exhibit: 

^ • THE SURVEYS 

The closure and follc^^rup surveys in their standardized form appear 
in the following pages. As mentioned previously (Instrument Development) 
in order to gather consistent data acrbss states it is essential that all 
stales ask the same questions using the same wording.' ♦ 

However, some states may wish to include additional questions relating 
to their own internal program planning and evaluation needs. Any additional 
questions that your agency may wish tx> add must be added to the end of the 
surveys following the standardized questions. This exposes all clients to 
the standardized questions in the same order, thus assuring comparable data. 
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Client Closure SurVev 



(Column #) 



■ (1-3) 
(4) 
(5-14) 



(15) 



<16) 



(17) 



Questionnaire No. I I I I 
U (Card Number) 
I I I I I I I I I I I 



(Client I.Q.) 



VR CLIENT Cin.SIIRF .qilRVFY. 

1. Are you satisfied with your overall experience 
with the rehabilitation program? [PLEASE CHECK 
ONE] 

1. Yes 

2. No I 

> 9- Not sure or no* opinion 

2. Are you satisfied with your counselor's per- ) 
formance (that is. did he/she do a good job^ 
for you)? [PLEASE CHECK ONE] 

1. Yes 

. 2. No 

9- Not sure or no opinion 

3. Did your counselor arrange for you to have 
physical restoration ?;prvirpc;. such as medical 
treatment, physical therapy/artificial limbs^' 
eyeglasses; dentures, hearing aids, etc.? 

. [PLEASE CHECK ONE] . / 

: 1. Yes 

2. No • 

9- I don't. remember 
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(Column ir) 










If YES^ are vou satisifed with these ser- 






vi'ceS? [PLEASE -CHECK ONE] 


(18) 










\ 2. Mo 


* 




9- Not sure or no opinion 

* 




' 5. 


Did vOur counselor arrange Vor you to have 






job training? [PLEASE CHECK ONE] 


(19) 




-1. Yes 






2. No • 






9- I don't remember 


S 4 


' 6 

• 


If YES. are" vou satisfied with the kind of 






training you received? [PLEASE CHECK ONE] 


. (20) 




1. Yes V 






2. No . 






9- Not sure .or no opiliion 




1 7 , 


Did vour counselor helD vou look^^r a Job? 






[PLEASE' CHECK ONE] 


(21)k^ 


r 


Yes "''rp^'^'-''- 






2. No 






9. I don't remember 

' » 




' 8. 


If YES^ are you satisfied with the help- you 






received? [PLEASE CHECK ONE]' 


' (2-2) 




__ 1. Yes • 






2. No 






9. Not sure or no opinion 



V 
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9. 



Were the services or training you received 
from the rehabilitation program useful in ^ 
helping you to perform in your present situ- 
ation or in helping you getyit? [PLEASE 
CHECK ONE] ( 

1. Yes • " 

2. No 

— 3. I received no services or training 
from the rehabilitation' program ' 
9. I have no opinion 
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client Fol low-Up Survey 



(Column #) 
(1^ 
(4) 
(5-14) 



(15) 



(16-19) 



Questionnaire No. /'/ I I I I 

I I (-Card Number) 

/ I I I I I IJ I I I (Client I.D.) 



2. 



VR CLIENT F0LL6W-UP SURVEY 

Which of the following statements best de- 
scribes your present work situation? [PLEASE 
CHECK ONLY ONE] 

1. I earn a wage or salary, either at a 

regular job or from self-employps^it 

2. I earn a wage or sjaldry in a sheltered 

worKshop or Business Enterprise Pro- 
gram (BEP) ■ / 

3. I am a homfemaker ^ 

4. I work in a family farm or business 

without pay ■ . 

,5. I am not working at present 

6. other (explain): - 



How much total income, if any. did you and 
your dependents receive last month from all 
sources of public welfare? [PLEASE CHECK 
ONLY ONE AND FILL IN THE SPACE] 

_ We received $_ last month> 

None^ 

I don't remember^ 
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(Golumn it) 



(20-21) 



(22-26) 



' 3. What were your total earnings last week (from 
a job. self -employment, sheltered workshop, 
or Business Enterprise Program (BEP)? 

— I earned $ J last week 

— I am working but I don't receive a 
wage or salary 

^ I am not working^ > 
. I don't know^ ^ 

4. What wgs your income last month from private 
- sources other than the earnings reported in ' 
Qwestion 3 (for example, from rents, divi-' 
dends. or private insur(5nc*e)? 

— .1 received $ . last month 

None-^ ^ 

I don't remember^ 



5. (Items assessing' functional ability and life 
status; items to be determined throi/gh pre- 
test of the Life Functioning Index (LFI). 



^These responses will require multicolumn codes (e.g., OOoS"'fo?~"None" 
on Question 2, 9999 for "Ldort't remember" on question 2). ^ 
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Vocatiojial Rehabilitation Program 
Staijdajrds Evaluation System 



. Ti'ainee Handbook-D: 
The Five Procedural Standards 



tpQ 



3200 odeliOG street be/keley. califania 94703 



. THE FIVE PROCEDURAL , STA^ffiARDS 
' OVERVIEW • ^~ ■-■ 

— ' nr; * ^ ' ^ 

The Procedural Standards coi^ist^of five goal -statements for the Voca- 
• ■tioMl Rehabilitation program., 'pertaining to R-30^aliditi, compliance with 
key regulations, and' certain aspacts of case handling. 'These standards 
-include: . . 

f. Standar<i^9 : Information collected on clients by the' R-300 

I _ '• . and all^^ reporting system? used by RSA shall be valid, 
reliably accurate, and complete. 

Standard 10 : .Eligibility decisions shall be based on accurate 
and sufficient diagnostic^ information, and VR shall continUalLy 
review and evaluate eligibilij^ decisions to ensure that deci- 
sions are being .made in a^rdance, with laws and regulations.- 



Standard 11 : VR shall ensure th^ eligibility decisions and 

jfoc 



client movement through the VR plocess occur in a timely manner 
. appropriate to , the needs and capabilities, of the clients. 

• Standard 12 : VR sh^ll provide an Individualized Written Rebab- 
. * ill tat ion^Pro gram for each applicable client, and VR and the 

client shaU be accountable' to each other for complying with 
'this agreement. 

• Standard 15 : Counselors shaH: make an effort to set realistic, 
goals ^for clients. Comprehensive consideration must be given 
to all factors in developing appropriate vocational goals' such 
that there is a maximum of correspondence between goals and 
outcomes: competitive goals should have competitive outcomes 

. ^nd noncompetitive pals should have noncompetitive outcomes. * 

The Procedural >Standards will be' reported for a given state agengy 
every third fiscal year. . RSA will conduct the dafta collection and will 



Jhe R-300 is the dat^ system which has been used by VR agencies and 
thejrefSre served as the framework around whieh specific standards were 
constructed. Changfes in the R-300 system may result in a need to alter 
certain data elements or instructions. ' ' , 
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report the results to^ach state agency. The data elements for each standard 
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consists of a number of individual information items pertaining to various 
aspects of the issues addressed by a given Procedural Standard. For example, 
the eligibility standard includes data elements on the process followed in 
declaring applicants ineligible, as wefl as data elements on the process fol- 
lowed for eligible applicants.. Thus, ftSA and state agency program managers 
will be presented (with information on ''how things are done" in the agency, 
with respect to the key processes embodied .in each Procedural Standard. It* 
is intended for states to use the Procedural Standards to benefit their pro- 
gram evaluation efforts and facilitate the improvement of services to clients. 
The information Obtained via the Procedural Standards ^ will form the basis for 
agencies to make appropriate changes in practices, where current processes 
are not in keeping with client interests and positive program performance. 

The methodology for implementing the Procedural Standards reflects RSA's 
desire to allow maximum flexibility to states in the VR process, yet still 
ensure attention to a common set of concerns and provide sufficient data to 
allow for program-wide analysis of these concerns. Ideally, a uniform pro- 
cedure would be followed by all states for monitoring these process areas, 
even though states retain differences in the ways ^they organize and conduct 
case service delivery* For example, indicators of compliance with legal 
requirements, such as eligibility and IWRP, should be the same for all states. 
All states should be asked a standard set of questions, with the resulting 
information being reported in d uniform manner. 

Most of the Procedural Standards* data needs are best met through a 
careful review. Thus, a single case 'review process. will be implemented to 
address the case review needs of four of the Procedural Standards. This 
process will use the Case Review Sche4ule (CRS) developed by the San Diegq 
State RCEP IX as the basic document for Procedural Standards data collection. 
The CRS has already been mandated by the RSA as the standardized instrument 
to be used by regional RSA offices whenever they conduct case, reviews . For 
Procedural Standards 10 (Eligibility) and 12 (IWRP), the CRS items essential 
to adequately assess compliance have been selected. These items make up the 
Modified CRS, which is considerably shorter than the full CRS. RSA .could 

\ 
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choose either the CRS or the' MCRS the^ instrument for collectrHg Procedural 
Stajidards data. - . • < ' 

. IVhile tie CRS is an appropriate vehicle for collecting compliance data, 
it Ijacks certain items needed to assess the validity of R-300 data (Standard 
9) or to assess' timeliness of case service {Standard 11).' For these stand- 
ards,, two separate- instruments have been developed to complement the CRS. 
These two instruments are incorporated directly into the CRS to provide a 
unified data collectipn instrument. 

Finally, Standard 13 on the correspondence between I5VRP occupational 
goals and final outcome uses data from the R-300. Because the R-300 is the 
sole data source for this standard, progress on this standard can be reported 
annually. 

To summarize, the Procedural Standards consist of five process-oriented 
goal-statements for the VR program. For Standards 9 - 12, data, collection 
will occur in a given state agency every third year, with RSA conducting the 
necessary- case reviews. Standard 13 uses R-300 data, and could be reported 
annually. 
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THE FIVE PROCEDURAL STANDARDS 



STANDARD 9 : R-300 VALIDITY 



Information collected on clients by the and all data * 

reporting systems used I>y RSAr„j5hall be valid. Tellable, ' ^ . 
accurate, and CDmpJ>ete. .>'V i. ' ' - ' ' ' " 

IWiile the VR service delivery systems need an objective data base from 
which to measure performance, inconsistencies and errors in reporting cur- 
rently exist among and within VR program data systems* , Similarly^ confusion 
or misunderstandings over definitions exist. This Procedural Standard 
addresses these shortcomings by ensuring that state agencies maintain accept- 
j^e levels of validity and reliability in their reporting of R-300 and other 
data. This standard also assumes that careful attention to good data proces- 
sing is pertinent to all of the standards. Thus,' given the import^i|pe of 
reliable, valid, and accurate data on which to base, the program's evaluation . 
capacity, this Procedural Standard relates ta the broad RSA goals of compli- 
ance, quality, and cost-effectiveness. 

Reliability, accuracy, and completeness of data could be checked- in< 
several , ways. UTiile a state agency could conduct validity studies on a 
periodic basis, and edit checks as a part of toutine data processings this 
standard encompasses a specific recoinmended procedure for states to follow 
to ensure the accuracy of data recorded and submitted to RSA through the 
R-300. Primarily, the case review, process includes an accuracy check between 
the case folder information, the R-3q0 fo™ itself, and, if th^tate has a 
computer system, computer output listing of R-300 items selected for revi.ew. 
In particular, those R-3D0 data items which are^used in computing the stand- 
ard 's data elements are subjected to checks of accuracy and validity through 
case foltler documentation. . ^ \ 
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This^ standard uses the R-300 Verification Instrument (Section I.C of 
the Modified Ca$e Review Schedule) as* it^ data source-, Table 2 shows the 
R-300 items wh'ich are checked using the R-300 Verification Instrument. 



Table 2 

R-300 Items Checked Using the R^300 Verification Instrument^.' 



Referral date 
Closure date 
Social Security number » 
SSDI status at referral 
SSI status at referral 



Major (primary) disabling condition 

Secondary disability 

Work statys at referral 

Earnings the week prior. to referral 



Receipt or nonreceipt of public 
assistance at referral 

Type(s) of public assistance re- 
ceived at referral (SSDI, SSI- 
Aged, SSI-Blind, SSI-^Disabled, 
APDC, Other) 

Monthly amount of public assistance 
receiyed at referral . 

Lehgth of time,, prior to referral , 
during which the client received 
public assistance 

Appropriateness of iJ^e Federal 
Special Program-- Identification 
checks (TF, Vet, MAW, PO,' WIN, 
SEC4, SF, SD) 

SSDI status at closure 

SSI. status at closure 

Work status at closure 

Weekly earnings ^ closure 

Receipt or nonreceipt of public 
assistance at closure * 

Type(s) of public assistance re- 
ceived at closure (SSDI, SSI- 
Aged, SSI-BJind, SSI-Disabled, 
AFDC, Other) 



Monthly ajnount of public assistance 
received at closure 

Occupation at closure 

DOT code fof that occupation 

Outcome status! (08, 26, 28, 30) ' ^• 

Reason for nonmiabilitated closure 

Total cost of all. case services 

Total cost of all case services 
provided in rehabilitation 
facilities 

Total cost of case services charged 
to Social Security Trust Funds 

Total cost of case services charged 
-to Supplemental Security Income 
Funds 

Receipt or nonreceipt and cost 
status of the following services; 
_.\ 

Diagnostic and evaluation; 

Restoration (physical or mental.); 

College or university; 

Other academic elementary or high 
school; 

Business school or college; 
Vocational schoal; 
On-the-job training; 
Personal and vocational adjustment ; 
tltscellaneous training; 
Maintenance; 
Other services i 

Services to other family members 



R-300 Verification Instrument appears in Section I.C^Jf the Modified 



Case Review Schedule. 
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STANDAjy) 10: ELIGIBILITY 

Eligibility decisions shall be based on accurate and 
sufficient diagnostic information, and VR shall con- 
tinually review ^^d, evaluate eligibility decl'sions to 
ensure that decisions are being made in accordance with 
la[ws and regulations ♦ ' ^ - 

The determination of an applicant's qualifications for eligibility is 
a critical point ii^.the VR process for both the client and the agency. 
This standard .seeks to protect the client's interests by requiring that 
state agencies install procediires for monitoring eligibility decisions in 
a sample o^^f cases. This system would ensure that all decisions are appro- 
priate, that* they are in compliance with legal requirements, and that they, 
are supported by the proper diagnostic information. Standard 10 pertains 
to two/broad RSA goals. First, inasmuch as the eligibility detennination 
proems rests on. a legal fcoting, the standard pertains to the goal of legis- 

ftive compliance. Second, it pertains to the goal of cost-effectiveness, 
since it is a misuse of money to serve ineligible persons,^ particularly if 
other, eligible clients are turned away due to an incorrect determination 
of eligibility. Thus^, a procedural standard for the review of eligibility 
determination implies Qoncem for the appropriateness of this decision-making 
^process. Information from this review will address two issues: (1) that 
clients who are not eligible for VR s^vices not be accepted for services, 
and (2) that clients who are eligible pre indeed accepted. ^ ' ' 

mile monitoring' ^nd review of eligibility decisions by supervising 
counselors or managers will provide ,a^j:*^cK on tfeat -xletermination, the actual 
procedures utilized in, providing this supervision will not be monitored. 
Consequently, states will be allowed ty Vetain flexibility in, establishing 
their monitoring practices. Although /cross-checks on impending eligibility' 
decisions ar,e important, they are not a requirement for this standard. 

The Modified Case Review Schedule (MCRS) serves as 'the data source for 
this standard. Table 3 shows the MCRS items used to address Standard 10; 
.the table is organized by the various relevant sections of the MCRS. 
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Table 3 - f 

Modified Case Review Schedule (MCRS) ttems Used 
for Standard 10 (Eligibility! 



SECTION I I; EVALUATION OF REHABILITATION POTENTIAL 

A; Preliminary Diagnostic Study - Status 02 

Does the preliminary diagnostic study,. , 

4. include an appraisal of the current general health status 
of the client? 

« 

5. include a psy.chiatric or psychological examination in all 
cases of mental or emotional disorder? 

6. include such examinations and' diagno3tic studies as necessary 
to; . ^ 

a, determine eligibility? ' 

b, determine the need for extended evaluation? 

7. place primary emphasis upon the determination of the client^s 
potejitial for achieving a vocational' goMl^ 

, ' • • 

8. support -the determination th^t the client has a medical ly-^ 
recognized physical or mental disability? 

9. support the determination that the medically-recognized - 
disability constitutes a 'substantial handi'cap to employ- 
ment for the client? , 

J 

. . 10. support the determination that VR services may reasonably • 
be expected t^^benefit the client in terms 'of employability? 

11. suppqrt the de'i-ermination that an extended' evaluation is 
necessary to determine that VR services may reasonably 
be expected tO;benefit the client in terms -of employafei-ldty' 



143 ' 



D-8 



Table 3 .(continued) 



SECTION II: EVALUATION OF REHABILITATION POTENTIAL 
B: Extended Evaluation r Status 06 



Does the case record , ^ 

14. contain a certification for extended evaluation to 
deterraijie rehabilitation potential? 

Does the IWRP for.' extended evaluation- (state form)... 

19. present the general basis for a determination that an . • 
extended evaluation of rehabilitation potential is nec- 
essary to make a determination of eligibility? 

show tTiat a thorough assessment of the client's progress' 
was made at least once in every "pO-day period during the 
provision of services under the extended evaluation? 



Does the case record... 

40. contain a Ce.rtification of eligibility for the continuance 
*of VR services? . 

42a^ show that the decision to terminate servicers was made in 
full consultation with* the client, or as appropriate, with 
the parent, guardian, or other representative? 

42d. show that the provision was made for a periodic review, at 
least annually, of the ineligibility decision? 
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Table 5 (continued) 



SECTION III; ELIGrSILITY - STATUS lo ~ ' ^ 

Does the certification of " eligibility, * ♦ 

. Ic. indicate that the. client has met the blsic eligibility 
rec^uirements? ^ * , 

How well does the counselor documentation in the cfS'e record... 

3. establish the presence of a physical or mental disability 
with necessary medical, psychiatric, psydhological? and 
other information? 

7. show that the substantial handicap to employment exists 
even though the client is employed, because the client ' 
IS unable to. obtain a gainful oc-upation .consistent with 
the client's capacities and .abilities? 

8. show the likelihood of .VR services enabling the client to 

achieve vocatioiml goals consistent with the client's 
capacities and abilities? 
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iable 3 (continued) 



SECTION VI rr •termination, of gases •- STATUS 08 

(Ntirabers in parentk«i5fes indicate Case Review item numbers) 



(2) 
(S) 



CASE CLOSED STATUS 08 FROM 00/02 - INTjfeVENING REASONS 

Does the , case record... . - 
document specific ''reasons for the closure action? 

show that the client, or appropriate ^ the parent, guardian^ or other representative ^ was Advised 
of the reasons for closure and the clo;?ure action taken? * , ^^.^^ 



CASE CLOSED STATUS 08 FRaM 00/02 - INELIGIBILITY 



Does the certification of ineligibility.. , - ^ 
(6a) indicate the date of certification? 

(6c) include the reasons for the determination of ineligibility? 

Does the case record... 

(7) . show that the client does not have a medically recognized physical or mental disability? ^ 

(8) show thsti the client does not' have a substantial ^handicap to employment? 

(9) show that beyond any reasonable doubt the client is not expected to benefit in^erms of employability 
fro;n VR services?* 

contain data supporting the ineligibility determination, in^^ding: 

(10a) ^ summary of medical and other case data obtained during the preliminary diagnostic study? 

- ♦ i i 

(lOd) documentation of a review of the ineligibility determination not later, thaii 12 months following 

such determination j | . • 

(11) show that the ineligibility determination was made only after full consultation with the client, or as 
appropriate, with the parent^ guardian, or other representative? I / 

^ - 1 / 

(12) document that the client was notified in writing of the closure action tak<jn? , 

document that the client was informed in writing of client i^ights and remedies^ including :^ 

(13a) . the right to adminisrative review and fair hearing? / 

(13c) the right to participate in the annual review of the ineligibility determination. 

(14) document any action^ and decision involving the client's request for an administrative review of 

agencyjafction or fair hearing? 

CASE CLOSED STATUS 08 FRaM 06 - INELIGIBILITY " " * 

Does the certification of ineligibility... 
(17a) ' ' indicate the date of certification? 

(17c) * include the reasons for the determination of ineligibility?- 

Does the case record... ^ 

(18) show that beyond any reasonable doubt ^the client cannot be e.xpected to benefit irt terms of employabilit) 

from VR services? ' , " * - . | ' * 

(20) / contain the rationale for the ineligibility determination as an amendment to the pVRP? 

(21) " show that the ineligibility determination was made only after full consultation with the client, or as 

apprtpipriate with ,the parent, guardian, 'or other representative? \ 

' ' . * i 

(23) document ''that the client wa^ informed in writing of the closure action taken? 

(24) document any action and decision involving the client's request for an a^dministrative review of ^ 
agency action or fair hearing? ^ ' ' \, 

(25) docume;it that the ineligibility determination was reviewed not later than 12 monthjs following such 
determination' ; \ » - 
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Table- 3 (continued) 



SECTION Vn ; ' TERMINATION OF CASES - STATUSES 30 and 28 



Does the certification of ineligibility... 
. 28a. infdicate the date of certification? 
28c. include the reasons ^for the determination of ineligibility! 

Does the'ca'se record... 

34. contain the rationale for the ineligibxlitj^ determination 
as an., amg-hdlnent to the program? 

. 35. show that thq ineligibility determination was made only 

after full consultation with the client, or as appropriate,, 
with the parent, guardian, or other representative? - 

3*6. document that the client was notified in writing of the 
•closure action taken? 




Does 



the case record show that the client was informed in writing of. 

37a, the right to administrative review and fair hearing? 

37b; the. right to participate in the annual review of 
the ineligibility determination? 



Does the case record.. • 

38. document any action and decision* invplvij^ the client's 
- request for an administrative review of agency action 

-or fair Hearing? ^ 

39. document that tSe determination that the client was no 
longer eligible was .reviewed not later ^than 12 months 

following such determination? 
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STANDARD 11: TIMELINESS ^ . - ' , . * . 

, ' ♦ / VR shall ensure that eligibility decisions and client - 

' movement through the VR process occur iji a timely mariner 

*^ appropriate to the needs and capabilities of the clients . 

Tills standard seeks to avoid delays in the*VR process that^re likely 
to 'impede' or hinder successful rehabilitation of the client. Rather than 
set a performancje standard using time-in-status to'define "undue delay," 
this Trocedural Standard ^yeqiiires that each state have a monitoring or flag-^ 
ing mechanism .for cases remaining in statuses over a given length of time, 
and a procedure. to evaluate the appropriateness of any case delay. Many of 
'.S(hp state VR agencies already have variations of "such a system in place. 

This stand^d pertains to the RSA goal of providing quality C2tse ser- 
vices, for two reason^. "First, one aspect of the quality of a client's 
service experience is the speed with which' his- or her case is handled. The 
client's perception of his/her treatment VR'>^n have an impact on his/her 
attitude toward. VR and about the ul'fefa^ess of participation in VR. Second, 
research on successful rehabilitation outcomes has suggested a relationship 
between timeliness and success, a relationship tokt may Se a consequence of ^ 
the client's perceptions as%liscussed above. . y ^ 

^ The issue of timely case movement or "undue delays" has been one of 
long discussion and Controveifsy. IVhiljs there is Hterature to support the 
correspondence between the time it takes for. certain processes, such'as the . 
eligibility decision, to take place and outcome, there have also been- ques- 
tions about interrater reliability in -the use of case reviews to judge time- 
liness. Nevertheless , *an overall review of timely case movement oh a 
client-by-client basis is best handled through casjs review, if items can be 
identified which have .good 'interrater reliability. 

Much effort has gone into establishing standards for the timeliness of 
casp service progress. Attempts to monitor the timeliness of service pro- 
' vision by way of a standard on "undue delay/* however, have, been hampered 
by several problems." The Hrst is that "undue delay" means different things 
to different people. IVhile the c^^rrent standards use the approach of arbi-* 
trary time periods to define "timely'* cas;e movement - (i .e . , eight months for 
timely eligibility decisions; 22 months for timely completion of the VR 
process, etc.), this approach has been widely criticized for'its la'ck of 
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^sensitivity to the legitimate differences in individual cases, A complex 
- case, perhaps involving' long-term educational ^services, might well require 
more than 22 mpnths, but may not constitute. an unnecessary <lelay in the 
service process . ^ 

In response to this criticism, other approaches to objective .measure- 
ment have been undertaken. For example, recording planned initiation and ^ 
completion d^tes for each service and monitoring compliance with' the Schedule 
has also been considered as a way to obtain all the information necessary for 
a careful' analysis of the timeliness of case movement with^|fedentifyS;ig a~ 
particular time peri-od as more or 'less appropriate. ' This ajBrnative, Myv 
ever, has been critized'for its cumbersome and time-consuming implement at io;i 
'^;^roc^s. ^ ^ ' ' ! » 

J .Subjective judgements of timeliness, while allowing for the reviewer to 
assess each case on its individual merits, have been vulnerable to criticisms 
of unreliability in application. Case reviewers might v^ell differ in judg- 
ments as to the cause^of a delay and,* therefore, differ in their interpreta- 
tion of whether the agency shouljM?e*hWd accountable for the. delay. One ■ 
reviewer might view a delay in the handling of a client's case as unnecessary 
and, therefore, the responsibility of the agency in question, while another 
reviewet might perceive the delay as stemming from a lack of client motiva- 
tion or actions by an outside vendor and, therefore, not the agency's respon- 
sibility. To correct these problems, a new tii?ieliness assessment instrument 
h^ been developed whiehr^J^ibile relying upon reviewer judgment, divides case 
assessments of timeliness intb two segments: first, a notation of whethet a 
delay has occurred in terms of time lapse between necessary activities; and 
second, an a^^^sment of the /reasons for the lapse. The relevant questions 
are included in \he MCRS an/ concern^ critical phas'es of case progress -- 
eligibility determiJftttrtrtC development of seirvice plan, service delivery, 
and termination. In addition, the Timeliness Assessment instrUme>Bf allows 
for notation of w^iethera case (?as 'handled with '^undue speed":, that is, if 
the case^moved too fast, in the reviewer^ s judgment', given the circumstances 
of me cas^. . • . 

The 'Timeliness Assessment Instrument is included as Section VIII of the 
Modified Case Review Schedule (MCRS). Table 4 shows the information items 
obtained for each reviewed case by the Timeliness Assessment Instrument. 



. ^ Table 4- , ^ 

Information Jiems Obtained by the Timeliness 
' A3sesgment Insti:iunent for Reviewed Gases^ 



1, Was the case handi^d' in a timely manner (i.e., without undue 
speed or undue delay)? • ' ^ ' ' . * 

2* If undue speed: , '* ' / 

a* Reasons for judging, the case as moving foo fast* 



3. If undue delay: ^ . 

a. Were the reasons for .delay documented in the clients^ 
case record? ' . , ; 

b/ Reasons for delay* 



^e Timeliness Assesstnept Instrum^t appears in Section VIII of the Modified 
Case Review Schedule. *^ > * 
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STANDARD 12 : IMP 



VR shall provide an Individualized Written Rehabilitatio n 
Program fo r' each applicable -client, 'and VR and the client 
shall be accountable to each other for complyi ng with this 



agreement 



• Several aspects of the Individualized Written Rehabilitation Program 
. ■' are addressed in this Procedural Standard, including: (a) compliance with 
. the requirement that an IWl^P- be fully developed for clients accepted for 
services or extended evaluation; (b) assurance of the protection of client 
y*^ights and^ client awareness of the remedies available for mitigating dis- 
satisfaction; (c) joint client/counselor development of the job goal and 
the service plan; (d) mutual client/counselor responsibility for follow- 
through on the agreement and annual review of its progress and appropriate- 
ness; and (e) the appropriate handling of plan revisions. 

Tins standard bears a relation to the RSA goals of ensuring compliance 
and cftKplity case services. Obviously, given the regulations mandating pro- 
vision of an IIVRP to aW accepted clients, this standard's relation to the 
^ . compliance goal^is clear. IVhile the regulations concerning the IWRP stip- 
ulate compliance' with- the provisions of the law, elevating the issue to the 
levefof a procedural standard will ensure compliance with .the legislative 
intent of the Ilv'Rf^. » 

Inclusion of this standard could be justified simply on the '.basis of 
■ the sti{ong regulation regarding compliance with the IWRP provisions oj^ the 
1973 Rehabilitation Act. However, perhaps an even more- important reason to 
include this standard is the fact that research has shown a positive assoc- 
iation between compliance with the IMP requirements and successful outcomes 
of the VR process. Since research has supported the premises underpinning 
the IKRP by showing that the process .and the possession of the IWRP affect 
client outcomes positively, a.dherence to the IWRP requirements becomes a 
powerful norm.for quality case management in VR, as well as a protection of 
client interests and rights. ' • ' 

The Modified Case Review Schedule serves as the data source for this 
standard. Table 5 shows the hiCRS items used" to address Standard 12. 
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Table 5 

Modified Case Review Schedule (MCRS) Items Used 
for Standd||^12 (IIVRP) 



SECTION II: EVALUATION OF REHABILITATION POTENTIAL 
B: Extended Evaluation - Status 06 

18. Is there an UTOP for extended evaluation jji the case record? 



Does the IWRP for extended evaluation (state form)... 

19. " p^sent the general basis for a determination that an 

extended evaluation of rehabilitation potential is 
necessary to make a determination of eligibility? 

20. Set forth the terms and conditions for the provision of 
service, including: 

a. client responsibilities in carrying out the 
program, such as attendance, . cooperation, 'etc.? 

b. the extent of client participation in the cost 
^ of services? 

21. document that the client was informed of client rights and 
remedies, including: 

^ a. the right to be fully consulted regarding 

any changes or amendments in the rehabilitation 
program? 

b. the right to administrative review in case of 
dissatisfaction with services? 

d. the right to participate in the annual review of 
the program? 

e. the right to participate in the artnual review of 
the ineligibility decision? > 



22. reflect that the IWRP for extended evaluation was maintained 
as a separate part. of the case record? 

25. show that the client received a copy of the IWRP and sub- 
staintial amendments?. • • . " « 

26. indicate that the program was developed and amei^ded with 
the client's participation, or as appropriate, with the 
parent, vguardian, ot other representative?;.; 

28. state the intermediate rehabilitation objectives? ^ ■ 
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. Table 5 (continued) 



SECTION JI.B (continued) 

Does the IWRP for extended evaluation (state form)... 

29. state the Vft services to be provided which are necessary 
• for the determination of rehabilitation potential? 

30. contain the projected date for the initiation of each 
servi ce? 

31. contain the anticipated duration for each service planned? 

32. provide the projected time within which rehabilitation 
objectives may be achieved? 

33. show that a thorough assessment of the client's progress 
was made at least once in every 90-day period during the 
provision of services under the extended evaluation? 

34. state the objective criteria upon which an evaluation of 
the client's progress is based? * 

.35, state the procedure by which the client is evaluated? " 

*-»96. contain a schedule for the periodic review and progress 
evaluation? 

37. contain a record of the results of scheduled reviews and • 
progress evaluations? 

38. show that a formal, annual review has been conducted if the 
IWRP has achieved at least first anniversary status? 

39. docuir^nt the client's views, or,, as appropriate, the views 
ot the parent, guardian, or other representative concerning ' 
the objectives and VR services beipg provided? 

Does the case record... 

42a. ,show that the decision to terminate services was made in 
full consultation with the client, or as appropriate, with 
the parent, guardian, or other representative? 

42b. show that the rationale for the decision to terminate 
services was recorded as a certified amendment to the 
IWRP for extended evaluation? 

42c. show 'that a certification of ineligibility was then e.xecuted? 

42d. show that the provision was made for a periodic review, at 
least annually, of the ineligibility decision? 



Table 5 (continued) 

SECTION II >B (continued) ^ , 

Questions 43 through 51 have two parts: - ' 

Item B: Does the case record document that the service was planned 
for the client? 

item C: Does the case record document that th^ service was given to 
the client? 

Each item is asked in reference to the following services (keyed to 
CRS question numbers) ♦ 



43. 


Diagnostic and Related Services 


44. 


Counseling and Guidance . ^^/^ 


45. 


Physical Restoration ■ 


46. 


Mental Restoration 


47. 


Vocational and Other Training 


48. 


Maintenance 


49. 


Transporation 


50. 


Services to the Family 


51. 


Specialized Services for Blind, Deaf, Severe Disabilities.^ 


52^. 


Telecommunications 


53. 


Occijpational Licenses, Tools, Equipment 


54-. 


Other Goods and Services 
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SECTION V: INDIVIDUALIZED WRITTEN REHABILITATION PROGRAM 
STATUS 12 AND ABOVE / ' 

1. . Is there an IWRP in the case record? 



Does^he im? (state form)... 

2. present the general basis for a determination of eligibility? 

3. set foi;th the terms and .conditions for the provision of 
services, including: / ' 

client responsibilities in carrying out the 
. program, such as cooperation, attendance, .etc. ? . 

b. Jhe extent of client iJarticipation in the cost 
j/f services? 

4. document that the client was informed of client rights and 
remedies, incl.uding: , . i-ignts and 



J 



a. the right to be', fully consulted regarding any 
. changes or amendments in the rehabilitation 
.program? ^ 

^.b,' ; the right to administrative review in case of 
, dissatisfaction with services? 

d. the right to par,ticipate in the annual review of 
the program? 

e. the right ,to participate,, in, the annual review of 
the ineligibility deoisionA 

5. reflect that the IWRI^ wss .maintained as a separate part of 
the ease record? " * . ' ^ 

6. show that the cliefit received -a copy of the IWRP and 
substantial amendmehts? " 

^' ihf m"" f.^"" the_pi«gram was deV^'loped ana amended with 
the client's participation o*^, aS^ -apprxjpriate with the 
parent, guardian, or. other repr^s^iitative?^ 

11. place primary emphasis on. the del|.rmination 'and achieve- 
ment of a vocational goal? ■ k-- 

. "'/-X ■ ., 

12. state the long-range employment ^go4i? . .,' . " '• ' 

13. state 'tie intermediate .rehabilitation objectives? 
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Table 5 (continued) 



I. • 



, SECTION V (continued) 

Does the LKK? (state form) 
14. 



15. 
16. 
17. 
18. 

19. 
20. 

21. 

22. 

23. 



state the specific VR serviced to be. provided to 
achieve the intennediate objectives and the employment 
goal? 

contain the projected date for the initiation of each 
service? 

contain the anticipated duration for each service 
planned? 

* ♦ * 

provide the projected time within which rehabilitation 
objectives and goals may be achieved? 



state 
the cl 
based? 



the objective /criteria upon which an evaluation 
ient's progreas^oward an employ ability goal is 



of 



state the procedure by which the client is evaluated? 

contain a schedule for the periodic reviews and progress 
evaluations? 



s^edi 



contain a record of the results of the sdieduled reviews 
and evaluations? 

show that a formal, annual review has been conducted, if 
the IWRP has achieved at least first anniversary- status? 

document the client's views, or as appropriate, the views 
of the parent, guardian, or other representative conceming 
the goals, objectives, and VR ser^ces being provided? . 



r 
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Table ^ 5 (continued) - 



SECTION V (continued) 



Does the case record,.. 



24a. show that the decision ^Mo terminate services was made in full 
consultation with the client or as appropriate, with, the 
parefft, guferdian, or other representative' 



>9 



24b. show that the rationale for the decision to termirfate services 
was recorded as a certified amendment to the IWRP? t 

24c. show that a certification of ineligibility was tl\en> 
executed? y 

24d. show that the provision was made for a periodic review, 
at least annually, of the ineligibility decision? 

25. contain a closure statement as an amendment to the program 
for a case closed rehabilitated? 

Does' the closure statement... , ' 

_25a. a description of the. basis upon which the client was 
' detejrmined to be rehabilitated? ' 



\ 26. Is there an amended IWRP for Post Employment Services' 



! ' ■ 

r 

t 
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Table 5 (continued) ^ ^ ; ^ 


SECTION VI: 


DELIVERY OF SERVICES - STATUSES 14, 16, 18-, 20, 22 and 32 


Questions 1- 


14 have twQ.jp^ts: * - 


Item B: Does the case xecord document that the service was planned 
for the client? p 

Item C: Does the case record document that the service was given 
to the client? 


Each item is asked in reference to the following services-^ (keyed to 
CRS question numbers) 1 ^ 


Services: 


i: 


Evaluation and Diagnostic Services 


2. 


Counseling and Guidance 


3. 


Physical Restoration 


4. 


Mental Restoration 


5. 


"^Vocational anfl Other* Training 


6. 


•Maintenance ^ ^ 




' i ■ ■ ■ ■ 

Transportation - 


8. 


Services to the Family 


9. 


Specialized Services for Blind, Deaf, Severe Disabilities 


/I.O. 


Te lecommuni cations . 


^ ■ 11. 


OccupationaT Licenses, Tools, Equipment 


12. 


Other Goods and Services 


13. 


Placement 


14. 


Po-s t - Emp 1 oymen t 




Table 5 ; ,(cQntinued)- 
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SBCrm VI i: TERMINATION OF. CASES--. ST ATUSES 30 arid 28 ' •• ' ' 
- ^ : ^ ' T] ; !— ' — : • i 

Does the case record. 1^ ' . . " '. ' ■ 

34.'- contairi jshe' ration^le' for. the "ineaigibllity determination 
as an amendment to the pr9gram? '.' '■• ' ,. . 

■•>^35: show that the ineligibility determination was made only 
•1 : ^f^®^ ^"^^ consurta,tion w.ith; the; clie:nt, or as appropriate, 
• ^ with, ithe pareftt, guardian, o?? tJther- representative?^ . 



J. 



Tins IS the same working as. used in item 20 (Tiot shown) pertaining to 
tlients closed 08 fram 06; except* that the work "IWRP" is substituted 
for '^program." 

This is the same wording as used^in Item 22 (not shown) pertaining , to 
clients closed 08' from 06 ^6 

/ 
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STANDARD 13: GOAL PLANl^IING 

Counselors shall make an effort to set realistic goals - 
for clients ♦ Comprehensive consideration >must be given 

' to all factors in developing appropriate vocational goals 
such that .there is a maximum of correspondence between 

/ goals and outcomes; competitive goals should have compet- 
itive outcomes and noncompetitive gpals should have non- 
competitive outcomes^* 

Competitive employment may not be the appropriate placement for all 
clients. Nevertheless, VR regulations require that all placements be into 
'•gainful activity'^ and that placements be consistent with the clients' 
"capacities and abilities," whether in competitive^ sheltered, or non- 
competitive employment. 

There is much speculation in the field over the abuse of "homemaker" 
and "unpaid family worker" categories, specifically regarding the use of 
these categories to ensure success rather than because the placement is 
appropriate* IVhile maximizing the proportion of successful closures (as 
captured in the Performance Standard 3) *is important to VR, it does not 
enst^re that noncompetitive placements are suitable for the client. This 
standard addresses the concern that noncompetitive closure categories not 
be used to salvage "successes" for. clients wTio were unsuccessful in their 
planned competitive goals. " v 

However, this standard is not intended to lock counselors and their 
clients into the goals set out in the original IWRP. Such an effect would, 
be a misapplication of the IWRP process. The^IWRP is, intended to be a ' yj 
statement of a realistically attainable goal which, if necessary, can be 
modified for a variety of valid reasons as the client progresses through 
the VR process. ^ i • -yr 

As such, state agencies should not use %he standard to overemphasize . 
the iinportance outmatching the outcome to the goal.. Th^^s would sisrve as a 
disincentive to setting ambitious (i.e., competitive -emplaynient) goals in 
the original IWRP, and would reduce Jthe flexibility 6f the cotinselor in 
refining the goa^l*^^ response to a client's progress.."' Instead, the results 
should be used in conjunction with dafa on client cbaracteHstics and ser- 
vice| to investigate how counselors can be^pre effective in the task. of 
"fitting" clients' potentials to feasihrle outcomes* In this way> the stan- 
dard is used appropriately to facilitate 'effective goal-planning . 
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« standard 13 uses four variations on a common theme as data elements: 

(i) » 0:^26 closures with competitive goa.1 AND competitive 

outcome " - J ... - , . ' 

' # of 26 clcTsures ' ^ ~ 

(ii) . # of 26 closures with* competitive goal BUT noncompetitive 
outcome 

^ of 26 closures ~ ; '■ — 

(iii) ^ of 26 closures with noncompetitive goal AND noncompetitive 

V outcome > 

# of 26 closures I r 

(iv) # of 26 closures with noncompetitive goal BUT competitive 

outcome ' 

• ^ of 26 closures ' ' 

The RSA-300 provides the data necessary to address this standard and, 
consequently, it can be reported annually. 
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COMPUTING THE PROCEDURAL STANDARDS DATA ELEMEHTS . 

The data element^ for the Procedural . Standards consist^ for the most 
part, of individual information^ items pertaining to specific aspects of the 
standard in question. In the Procedural .Standards reports, these informa- 
tion items will be presented in terms of. a series of ''percentage achieved*'* 
scores; for example, the percent of reviewed case records which "document 
that the client was informed of client' rights and remedies, including the 
right to be fully consulted regarding any changes or amendments in the rehab- 
ilitation program" (MCRS,.Item V.4.a, used for, Standard 12).^ Likewise, all^ 
other it'ems from the MCRS will be computed and reported as a "percentage 
achieved" score. ^ With this, program managers will be able to see the extent 
to which an agency is in compliance (or has valid R-300 data, or serves its 
clients in a timely manner) in terms of a number of separate indicators. 
This will allow program managers to pinpoint specific problems occurring in 
the agency's case-handling and data-recording processes. 

The one exception to the "percentage achieved" method occurs on Standard 
13. As noted in the .discussion of that standard, its data elements consist 
of four similar ratios, each of which compare clients' IWRP, goals to their 
ultimate outcomes. 

Given the Straightforward interpretation of the Procedural Standards data 
elements, the instructions for computing the data elements can be stated simply: 

1) collect the necessary data; and 

2) compute the percentages, using valid cases only. 

The only remaining task is to specify the information items used for the 
Procedural Standards. Table 6 provides the specifications. The table lists 
the Procedural Standards (and, for Standard 13, the four data elements), the - 
data source and item specifications for the data items and instructions for 
completing the data items. ^ , • 



As with the Performance Standards data elements, the "percentage achieved" 
scores must be computed using valid cases only . In the example "given above, 
for instance, we would divide the number of cases for which the case record 
documented that the client had been "informed of rights and remedies regarding 
IWRP changes," by the number of clients reaching Status 12. AH other cases 
^re "invalid" (for this particular data element) and are not to be used in 
computing the percentage score. 

16Z 
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Table 6 

Summary of Stamianis, Data S ources, ami l):ita Snecificntion.. 
for the VR I'roRram Trocedural Standards 



Standard 



9. R-300 Validit j ^r 

Informatioiv collected on clients by the 'R- 300 
and all data reporting systems used by RSA 
shall be valid, reliable, accurate, and com-* 
plete. 



10. 



EJigibility 

Eligibility decisions shall be based on 
accurate and sufficient diagnostic informa- 
tion, and VR shall continually review and 
evaluate eligibility decisions to ensure 
that decisions are being made in accordance 
with laws and regulations. 



1 1 . Timel iness 



VR shall ensure that eligibility decisions 
and client movement through the VR pro- - 
cess occur in a tamely manner appropriate 
to the needs and capabilities of the 
clients. 

VR shall provide an Individualized Written 
Rehabilitation Program for 6^ch applicable 
client and VR and the client^shall be ac- 
countable to each other for Complying with 
this agreement. 



Data Source 



Modified 
Case Review 
Schedule 



Data Specifications 



Section I.C (R-SOO Verification 
Instrument) 



Modified 
Case Review 
Schedule 



Modified 
Case Review 
Schedule 



Modified . 
Case Review 
Schedule 



Section II. A. 
Section II. B. 
Section III: 
Section ,VII: 



4-11 

14, 19, 33, \0, 42a, 42d 
Ic", 3, 7, 8 

2, 5, 6a, 6c, 7, 8, 9, 10a, 
lOd, 11, 12, 13a, 13c, 14, 
17a, 17c, 18, 20, 21, 23,' 24, 
25, 28a, 28c, 34, 35^ 36^ 
37a, 37b, 38, 39 



Section^ VIII.A and VIII.B (Timeliness 
Assessment Instrdment) 



Section II. B: 



Section V: 



Section VI; 
Section VII: 



18, 19, 20^, 20b, 21a, 21b, 
21d, 2le, 22, ^3, 26, 28-39, 
42a, 42b, 42c, 42d, 43-54B. 
. 43-.54C 
1, 2, 3a, 3b, 4a, 4b, 4d, 4e, 
5, 6, 9, 11-23, 24a, 24b, 24c, 
24d, 25a, 26 
1-14B, 1.14C 
20, 22, 34, 35 
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'Table 6 (continued) 









Standard 


Data Source 


Data Specifications 



13. Goal Planning 

Counselors shall make an effort to set realis- 
tic goals for clients. Comprehensive consid*. 
oration must be given to all factors in 
developing appropriate vocational goals such 
. that there is a maximum of correspondence 
between goals and outcomes: competitive 
goals should* have competitive outcomes and 
noncompetitive goals should have noncompeti- 
tive oi|t cones, 

' Data Elements; ^^^X^ 
(i) of 26 closures with competitive 



goal AND competitive outcome 

if of 26 closures ~ 

(ii) * of 26 closures with competitive 
goal BUT noncompetitive outcome 

* of 26 closures 

(iii) # of 76 closures with noncompetitive 
goal AND nqncompetitive outcome 

* of 26 closures, 

(iv) # of 26. closures with noncompetitive 
goal BUT competitive outcome 

* of 26 closures 



RSA-300;. 

Wark Status 
of TOP Goal 



Work Status 
at Closure 

* 26 closures 



Item 3.B fCompetiti»/e empLoyinent^= codes 1 
(wage and salar^ workers) and 3 
(self-ejnployQd% not SEP)] 

Item 4,1 [Competitive employment « same 
as for Item 3.B) 

Item 4. P. 2 {total number) 
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im-RODUCTION TO THE MODIFIED CASE REVIEW SCHEDULE 
OVERVIEW ' • • . . ' I 

The Case l^eview Schedule (CR§;), designed by the Rehabilitation Counselpr 
Education Program (RCEP) at San. Diego State University, can be used to deter- 
mine if state VR programs axe providing services consistent with the federal 
regulations and guidelines mandated by the Rehabilitation Act of 1973. 
Berkeley Planning Associates, under contract to the Rehabilitation Services 
Administration to develop an evaluation standards system, used a modified 

« 

versidn of this case review instrument to assess selected procedures. 

Five of the 13 program standards are "process*' or ''procedural" in nature; 
of these, four relate to case reviews: - ' " - 

• Standard 9 : Information^ col :^eqted on clients by the R-300^ and ' - 
all data reporting systems used by RSA shall be valid, reliable, 
accuraj:.e',— and complete. ' ^ 

/ • Standard 10 : Eligibility decisions shall be based upon accurate 
and sufficient diagnostic information, and VR shall continually • 
review and evaluate eligibility decisions to ensure that deci- 
sions are being made in accordance- with laws and regulations. 

. • Standard \\ \ VR shall -ensure' that eligibility decisions and 
, client movement through the VR process occur in a timely manner 
appropriate to the needs and capabilities of the clients, 

• Standard 12 : VR shall provide an individualized written .rehab- 
ilitation program for each applicable client, "and VR and the 
client shall be accountable to each other for complying with 
this agreement. 

The fifth Procedural Standard, Standard 13, p(5rtains .to the setting of "real- 
istic" goals for'VR clients, consistent with their capabilities and abilities 



the R-300 is the data system which has be^n used, by VR agencies 'and 
therefore served as the framework ardunS which specific standards were con- 
structed. Changes in the R-300 system 'may result in a need to alter certain 
standards br instructions. 
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whether this means setting competitive employment goals or sheltered or non- 
competitive employment goals. The data elements necessary to' address this 
goal are included in the R-300 system and^ consequently will not be discussed 
*^ in this guide* A review of this standard and its corresponding data .elements 
is found in "An Overview to the Five Procedural Standards a companion docu- 
ment to this guide* " • — ~ - ———--r. 

The Case Review Schedule as modified (primarily item deletions) by BPA 
can be used, to verify agencies' compliance with Standards 10 and 12, tlie 
standards regarding eligibility and the IWRP. BPA designed two new s€fctions 
for inclusion in the Modified Case Review Schedule to address Procedural . 
Standards 9 and 11. These two additions are.^ 

• The R-500 Verification , designed, to assess the degree to which 
information subjnitted to RSA on critical items of the R-300 
(defined as items used in the calculation of Performance Stand- 
ards' data elements) was corroborated by casefile information, 
in order to respond to Standard- 9; and 

• The Timeliness Assessments, designed to link subjective assess- 

^ ) 

ments of the timeliness of case movement, to objective data on 

the length of tijne spent in variou^statuses by different dis- 
ability types, thereby generating a data pool from which 
accountable parameters for times-in-status could be drawn, in 
order to respond to Standard 11. ' 

IVhether administered by the states or by the Regional Rehabilitation 
Services Administration, the training of ' case reviewers will emphasize all 
aspects of the case review process to ensure consistency within states and. 
across states in collecting* the data. Only the smallest margin for self- 
interpretation of the Modified Case Revi^ Schedule will be necessary follow- 
ing the thorough training and explanation of each item oil the MCRS. By 
integrating the R-300 Verification and. f he Timeliness Assessment into the 
MCRS, the process of. examining case files can become more efficient. Also, 
the>lCRS includes only those iteips which directly relate to compliance, with 
eligibility and IWRP standards, !as well as items relating* to supportive eval- 
uation and, therefore, is the most compact' instrument possible for fully 
verifying compliance with Proce^ral Standards, and allowing for the investi- 

jm^it performance. . *^ < 

'. /I. ifi"'' 
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DATA ELBfENTS IN THE MODIFIED CASE REVIEW SCHEDULE ' 

As part of its project to revise the VR Program Standards, Berk'eley 
Plannifig Associates (BPA) reviewed the .Case Review Schedule (CRS) «deve loped 
by the San Diego State University RCEP IX, and selected those items needed 
. to adequately address Standards 10 (Eligibility) and 12 (IWRP). In addition, 
items were selected which viere felt to be of use in the problem identifica- 
tion stages of the decision-support system. ' Finally, BPA developed two. new 
instruments - the R-300 Verification Instrument and the Timeliness Assess- 
ment Instrument -- to address the Procedural Standards not already covered 
by the CRS. BPA merged those two new 'instruments with the items from Hke 
■ existing CRS - selected to address Standards 10 and 12, ijfd thg problem 
identification activity — to form a new Modified Case Review Schedule (MCRS) 
The MCRS serves as the unified data source for Standards 9 - 12'. Below, we 
briefly describe each of the sections of the MCRS, identifying -their specific 
use in the Procedural Standards and the information they elicit. A summary 
of the relationship of each section to each of four.televant standards is 
presented in Table 2. 

i 

Sections I. A and I.B: Identifying , Information and Significant Case Data 
> 

These sections provide- information to identify the client (e.gi, his or 
her case number) for use in analyzing the other MCRS data and for merging 
the data with other documents (such as the client's R-300). As well. Section 
I.B records certain singificant dates relevant to the client's program exper- 
ience (e.g., date of service initiation). These are used 4s supplemental 
information for Standards 9 - 12. ; ^ 

Section I.C: R-300 Vertification Instrument 

The R-300* Verification instrument .is designed to respond to Standard 9. 
It assesses thg degree to which information submitted to RSA on critical 
items of the R.-SOO was corro.borated by casefile information. All of the 
data items on the R-300 Verification instrument are necessary to ensure the 
integrity of the data source, on which many of the Performance Standards' 
data^l fluents, are based. One of the purposes of Standsird 9 is to verify the 
•R-300 information such that users of the R-300 data can have confidence in 



Table '2'. ' . • 
Summary Table of MCRS^ Data -gections 
And Their Relationship to Specific Standards 



MCRS Data Sections ' , 


Data for 
Standard 9 


Data fcrr 
Standard 10' 


if 

Dat^br 
.S£andafd 11 


D^ta for 
Standard 12 


Section I.A:^. Identifying Information 


• \ / 








Section I.B* : Significant Case Data 
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Section I.C: R-300 Versification Instrument 
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Section II. A: Rvaiuation Status 02 < 










Section II.B: Bxtended evaluation -? ^ 
Status 10 ' ; ' ^^-r^" '^ 




/ 
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Section III: Eligibility"'^r^tatus 10 ^ 










: / . 


Section IV: ^valuation Problem , 

Iclentification ' , ■ / 
■ ■ ^— '-^ L 






v,>>... 


/ 




Section V: . 'IWRP - Status '12 / ' ^ 

: »^ / 








■ / ■ ■ 


Section VI: ^ Delivery of Serviced ' 

' . ' Statuses X4, 16, 18y20,- 22', 25' 
~ 7^ ' ' 7^ ^r-^^^- 
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.Section VII: Termination of S^vices * ^' 




.'\ 


— 1 \ 
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Section VIII: Timliness A^^ssmdnt 

Instrumejvt^ ^ 











the accuracy of the data reported to the states, the department administra- 
tion, and the Congress, ' In the Procedural Standards, "verification of 
accuracy" refers to a manual confirmation procedure intended to ensure that 
file information. supports and corroborates the R-300 docum'erlts. 

Section II. A: Evaluation of Rehabilitation Potential : 
^ Preliminary Diagnostic Study -. Statu s 02 
m ' 

This sectionr used for Standard 10, assesses the extent to which the case 
record documents the occurrence of the various activities needed to conduct 
an effective preliminary diagnostic study. This study, completed during the 
application phase, should contain all of the information necessary to make a 
reasonable assessment of a clTent's. eligibility for VR services. Among these 
necessary pieces of information are all relevant medical and psychiatric 
examinations, and other evidence th.at supports 'the client's need and eligibility 
for rehabilitation services: Without this information, agencies will not be 
able to select the disabled individuais^ who^can most- benefit frJm available 
— but limited --,VR services. 

Section II. B: Evaluation of Rehabilitation Pot^nJ;4^ 
Extended Evaluation Status 06 

^Section 'II. B is used for Standards 10 and 12, and for problem identifi- 
cation. In regards to Standard 10 (Eligibility), Section II. B seiks docu- 
mentation that the state agency has followed proper procedure in jllacing- ' 
applicants into extended evaluation, status 06. In,parti<iUlar,>dib concerns 
are that case records include: 

a certification for extended evaluation to determine rehlbilli- 
tation potential;. 

the basis for the need for extended evaluation; 



evidence of the occurrence of thorough assessments of progress 
at least every 90. days; and 



• documentation of\,the eligibility decision resulting^-^rrjfn exten- 
ded evaluation. 

i 

Provision of this information helps ensui:e that extended evaluation is .used 
Only when appropriate, that the client moves through extended eva-luatidn| in 
a timely manner, and that the minimal recording needs foi^ Status 06 are Main- 
tained. 



/ ' . ' . V i * 

In regards to*Standard 12 (IWRP), this section seeks to document that. ' 
the IWRP's for clients placed into extended evaluation contain all'pf the 
information required under Status 06, In this context, 'the concerns" of .the 
Procedural Standards are that IWRP's for cases entering 06 do the following: 

• define the terms and conditions for the provision of services; 

• document that the client was informed of specific rights, 
including the right to participate in the development of the 
program; • - % 

• specify a vocational goal and' a timeframe for its achievement; 

• specify evaluation procedures and criteria; 

• document the final eligibility decision and; for those clients 
closed as ineligible:* • ' ' 

document that the client participated in the decision; 
' and 

^ document that provision was made for periodic review. 

Provision of this information helps ensure adherance to the IWRP provisions, 
and ,helps ensure that clients move through Status 06 in a timely manner and 
are av^are of their rights to continued services or review if declared inelig- 
ible. 



iv^are 



Section III: Eligibility'- Status 10 

IVhile it is important to document in the preliminary diagnostic study 
the extent to which applicants meet the basic eligibility criteria, it is 
even more importiant to ensure that all clients accepted for services meet 
all of the requirements for eligibility. \ The purpose of Section III is to 
demonst^te compliance with Standard 12 by documenting that a certification 
of eligibility was completed for 6ach accepted client, and that the case 
record confirms: ^ 

• the existence of a disability; *• 

• the existence of a substantial . handicap to employment; and 

• the likelihood that VR services will benefit the clients. 

0 

Section IV.: Evaluation of Rehabilitation Potential: : * ' ' 

* Thorough Diagnostic Study Status 0^ and 10 

This section is used solely for problem identi>^iCation, It includes 

questions on the quality and scope of the thorough diagnostic study. 
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Section V: Individualized Written Rehabilitation Program - Status 12 

•f ■ 

Section V is. used for Standard 12 (I^P) • In a sense, the purpose* of 
the IWRP is to establish an alliance between, the agency and the client for 
the provision of certain services toward the achievement of a. specif ic .voca- 
tional goal. As such, it is important that the IWRP contains all the 
information necessary to establish such an alliance. Section V documents, 
among other things: - 

• that the client was informed of the terms and conditions for 
the provision of services; ' . ' 

• that the client was informed of client rights;- 

• that the client participated in the full planning and review 
process; and 

• that the IWRP contains essential information such .as goals, 
time frames, evaluation procedures, and schedules, etc,^ 

inclusion of this information in the IWRP clarifies the roles, relation- 
f ships, and duties of agency and client toward achieving the vocational goal* 



This is the esse^5?e of the IWRP process- 
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Section VI: Delivery of Services - Statuses 14, 16, 18, 20, 22, and 32 

This section, used for Standard 12, complements the information provided 
in Section V. IVhen these two areas are reviewed together, they describe the 
overall VR process, consisting of the plan (i.e., the terms, conditions., and 
information set forth in the IWRP needed to provide services) and the specific 
program of services undertaken to achieve the vocational goal embodied in the 
IWRP. By knowing the extent to which planned services are actually delivered, 
we can determine the extent of effective "follow through** on the service plan- 
ning process, in keeping with the spirit of the IWRP legislation. 

Section VII: Termination of Cases ^. 

Section VII rebates to Standards 10 and 12. As might be expected given 
its focus on eligibility", the questiotis used for Standard 10 focus on non- 
successful closures: OS's (from both 02 and 06), 28^s, and 30's. these 
closure statuses, Standard 10 attempts to assess (through review of case 
record documentation) the following compliance issues: 

\ 
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• Does the case record document the ineligibility/termination ' 
decision, and the basis for that decision? ' ' ' . 

*• ^ Have- clients been granted their legal rights to^ participate in 
the ineligibility/termination decision? - ^ ' 

• Have clients been informed of their right to an annual revielv 
'of the decision? and ' 

• Have the required annual reviews occurred, and the results. 

been documented? - - • ' ; / ' * ' 

; r : ' ^ 

Th^ need for this information is twofold. First, it is reasonable- to 
expect that supervisory personnel might want' to review any .given case involv- 
ing ineligibility or unsuccessful termination for, any given^^unselor . 
Management personnel should have the ability to review cases samples af 
random (i.e,, across all coui^elors, as in ah audit-type procedure), or to. 
target reviews to particular counselors (as might be needed fdr less exper- 
ienced counselors). Either way, it follows that for any case of ineligibility 
or unsuccessful closure, the closure .action an.d 'the basis for it must be • 
adequately documented iji the case record. - ' / ' // 

The information in Section VII is important for a second reason, which 
stems from VR's desire .to protect the rights of its applicants and clients/ 
The best way to ensure such client protection is to require proof ^ in the 
case record that the necessary steps |^ave occurred: for example, a *'Bill of 
Rights" signed by. the client; a schedule for review, signed perhaps by the 
client; and ''Results of Review" form, which could be signed by the client. 

In short, VR^ agencies need to knpw the reasons for unsuccessful closures, 
arid need to ensure that the ineligibles and unsuccessful closures are aw^re 
of. their rights to review^ Once this is ensured, then informed clients^-- 
whose circumstances have changed such that they are eligible -- hopefully will 
'Reenter the system later and be successfully rehabilitated. . - 

In addition to its uses in assessing eligibility-determination processes. 
Section VII is' also usfed^tor Standard 12, on the IWRP. The question^ used 
he-re seek to ensure: ' - 

• that the rationale for closure decisions are recorded on» the ^ 
• TWRP; and ' ' - ^ 

• that the 'client (or his/her appropriate representative) was 
consulted prior to the -closure decision. , ^ 



\ 
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Provision of this information ensures that the agency has a source from 
which it can draw Information about past ineligibility decisions, to make ^ 
, ^ sure they were made in a consistent manner, and to ensure that? client rights 
/ ' * were protected during the closurB process. . 

/ Finally, several questions pertaining to 26 closures are included in 

; Section VII. These questions will be used for problem identification. 

. Section VIII: Timeliness" Assessment Instrument 

The Time'liness Assessment instrument responds directly to Standard\i. 
It is designed to link subjectiye assessments of the timeliness of cas^ 
movement to objective data qn the length of time spent in various statuses 
by different disability types. "The assessments are used in conjunction with 
data bn client characteristics and services provided, to investigate how 
agencies might avoid undue delays in the service process. 

SAMPLE SELECTION AND FREQUENCY OF REV IEW 

^ , As designed, the MCRS does not needf%o be completed on all individuals 

in a given VR system to determine "the level of compliance with each of the 
four Procedural Standai'ds represented in this system. Such a determination 
can be made based upon a review of a selected sample of cases. This selection 
process, however, needs to consider a number of issues in order to ensure that 
/ the sampled cases accurat^^ represent the experiences of the general popula- 

tion. Originally, it was thought that this representativeness could.be 
achieved by randomly selecting a number of cases from each of the potential 
closure categories. This strategy, however, did not result in a sufficient 
number of cases in certain classifications to allow for a full analysis of all 
of the issues identified in the" four standards. This problem can be corrected 
simply by incorporating an additional elemen;t into the sampling framework: 
specifically, the variable ''entered/did not enter extended evaluation** should 
be included as a sampling criterion and the individual selecting, the sample 
should be. careful to include a sufficient number of cases in which entry occur- 
red. Likewise, future data collection efforts sliould include OS's in sufficient 
numbers to allow assessment of compliance with the regulations pertaining to 
ineligibility determinations. 
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In addition to' including a sufficient number of cases from $ill relevant 
closure categories, it is- also beneficial to conduct case .reviews on a sample 
of those clients currently in-service* Initially, those cases s'elected for 
review in the pretest were cases for whom services had been terminated. In 
assessing this procedure, pretest reviewers noted that since the sample 
included only closed cases, there was no possibility for using the review 
. process as a management tool. To correct this short-coming, it is. recommended 
that agencies select cases for review that are at various points in the ser- 
vice system, including eligibility, extended evaluation, plan development, 
service provision, and closure. By selecting cases from the full range of 
stages in the service process, the MCRS can be used not only to analyze the 
issues surrounding successful and unsuccessful closures but also to assess 
current operating procedures and to offer the possibility of taking corrective 
"action in those cases where the MCRS indicate problems exist. 

The last point con.ceming conduct of the data collection relates to the 
physical location of the data collection .effort. Logistically, it is prefer- 
able to centralize the data collection activity in one place (e.g., the state's 
central offices) , However, it was noted during #the pretest that requiring 
removal of case files from the district offices was problematic in c^ses where 
the files were needed by caseworkers or clients. This problem will be further 
exacerbated by the suggestion that the sample also include in-service statuses. 
This issue should be considered and resolved as appropriate in each particular 
state.\ A possible solution might involve the reviewers traveling to inidividual 
districit offices -to .conduct reviews, if this can be done without incurring 
ex.cos$ive travel or other costs. 

Statewide assessments should occur -every three years, and never les^..' 
frequently 'than every four years. These statewide ^assessments should be 
supplemented by more frequent, targeted spot checks, as suggested by the 
statewide assessments. 

CASE RF.VIEWERS: QUALIFICATIONS AND EXPERIENCE 

The Procedural Standards rest squarely on the process of case review. 

The quality of those reviews, and their resulting data, is strongly influenced 

by the capabilities of the people who conduct the reviews. There are four 
main qualifications an effective ^case reviewer must have: 
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• The most important quality desired in a reviewer is that he/she . 
have an iittimate familiarity with a wide variety of aspects of state agency 
operations. The MCRS requires that reviewers possess detailed undei'standing 
of alj aspects of t-he program's operations. - * * ♦ . 

• In addition^ to having familiarity with the particular agency's over- 
all operations, ideally case reviewers would have experience in casework, f 
(i.e., c ounseling) .• casework supervision, overall operations, andy^dministra- 
tion. ' . • 

• The third quality is that of »»aloofness»^• the reviewer should have * 
no conflict of interest when reviewing cases. 

• Finally, the last qualification has less to do with' the personal 
qualifications of reviewers than with the way the reviewed positions will be 
structured. Ideally, the reviewing function wiU be one of the revie wers^ 
main assigned responsibilities . In other words, the reviewing task should 
not be shifted to different personnel with each data collection cycle. The 
benefits of maintaining a core staff of reviewers are numerous. First to 
the extent that the MCRS is part of a person's overall duties, perceptions 
that this type of data collection is intruding on the person's Qther regular 
duties will be minimized. Further, by assigning the review function to par- 
ticular individuals, the person's familiarity witlT'the task will be reinforced, 
his or her reviewing ability .will be improved, and the necessity for intensive 
training with each new data collection cycle will' be obviated. 
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